Most LLM testing focuses on whether the answer is right. Almost nobody tests whether the output is structurally reliable: does it actually conform to the schema you asked for, every single time? If you're piping model output into a pipeline, an API, or an agent, that question matters just as much as content accuracy. This post walks through a quick, practical way to measure it.

What "JSON mode" actually guarantees (and doesn't)

"JSON mode" is one of the most misunderstood features in LLM APIs. Turning it on tells the model to constrain its output to valid JSON syntax. That's it. It does not mean the output will match the shape you actually need.

In practice, a model in JSON mode can still hand you a document that parses perfectly and fails your schema in a dozen ways:

• A required field is missing entirely.
• A field that should be a number comes back as a string ("42" instead of 42).
• An enum field returns a value that's close but not exact ("Medium" instead of "medium").
• Extra, unrequested keys show up in the object.
• Nested objects are flattened, or arrays contain the wrong item type.

None of these break JSON parsing. All of them will break a strict consumer downstream, whether that's a database insert, a typed function call, or another service expecting a specific contract.

This is the core distinction QA needs to test for: syntactic validity (is it JSON?) versus schema conformance (is it your JSON?). Most teams only check the first, usually implicitly, because a JSON.parse() call not throwing an error feels like success. It isn't. It just means the model didn't produce garbage, not that it produced what you asked for.

The failure mode that makes this worth testing systematically is that conformance is rarely 0% or 100%. It's probabilistic. A prompt might conform correctly 98 times out of 100 and then quietly drop a field on the 99th, often for inputs that are longer, more ambiguous, or structurally unusual. That kind of intermittent failure is exactly what unit tests and manual spot checks tend to miss, and exactly what a small automated harness can catch.

That's what the next section builds: a way to run the same prompt repeatedly, validate every output against a real schema, and turn "seems to work" into a number you can track.

Building a conformance harness

The idea is simple: fire the same prompt at the model N times, run each response through a real schema validator, and record whether it passed or failed. No need for a testing framework beyond what you already have. A loop and a validator get you most of the way there.

First, define the schema you're actually testing against. This is the contract, not a description of what "good JSON" looks like in general, but the exact shape your downstream system expects, including which fields are required and which values are allowed.

                
from jsonschema import validate, ValidationError
from openai import OpenAI
import json

client = OpenAI()

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "priority": {"type": "string", "enum": ["low", "medium", "high"]},
        "tags": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["name", "priority", "tags"],
    "additionalProperties": False,
}
                

Next, wrap a single model call in a function that validates its own output. It should distinguish between two different failure types: invalid JSON (the response didn't even parse) and schema violations (it parsed fine but broke the contract). Keeping those separate matters for the breakdown in the next section.

                
PROMPT = "Extract the task details as JSON: 'Fix the login bug, it's urgent, tag it as backend and auth.'"

def run_once():
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": PROMPT}],
    )
    raw = response.choices[0].message.content
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return {"passed": False, "reason": "invalid_json", "raw": raw}

    try:
        validate(instance=parsed, schema=schema)
        return {"passed": True, "raw": raw}
    except ValidationError as e:
        return {"passed": False, "reason": e.message, "raw": raw}
                

Finally, run it N times and collect the results. Keep the prompt fixed across runs. You're measuring the model's consistency on one input, not prompt variability.

                
N = 50
results = [run_once() for _ in range(N)]
                

A few things worth keeping in mind, since they affect how trustworthy your results are:

• Don't dedupe or cache responses. Each call needs to hit the model fresh. Conformance failures are often intermittent, so caching defeats the point.
• Capture the raw output on failure, not just the fact that it failed. You'll want it for the breakdown in the next step, and for debugging edge cases later.
• N matters. 50 runs is a reasonable starting point for a smoke test, but if you're deciding whether to trust this in production, run it a few hundred times, ideally spread across different times of day, since some providers show drift depending on load.

At this point you've got a list of pass/fail results with reasons attached. The next step is turning that into a number you can actually act on.

Turning results into a conformance rate

A pile of pass/fail dictionaries isn't useful on its own. The point of running N trials is to compress them into a number you can track over time and compare across prompts, schema versions, or models. At minimum you want an overall conformance rate. Ideally you also want a breakdown by failure type, since "12% invalid JSON" and "12% wrong enum values" point to very different fixes.

                
from collections import Counter

def summarize(results):
    total = len(results)
    passed = sum(1 for r in results if r["passed"])
    failures = [r["reason"] for r in results if not r["passed"]]
    failure_breakdown = Counter(failures)

    return {
        "total_runs": total,
        "conformance_rate": round(passed / total * 100, 1),
        "failure_breakdown": dict(failure_breakdown),
    }

summary = summarize(results)
print(summary)
                

For the schema from the previous section, output might look like this:

                
{
    "total_runs": 50,
    "conformance_rate": 94.0,
    "failure_breakdown": {
        "'priority' is a required property": 2,
        "'High' is not one of ['low', 'medium', 'high']": 1
    }
}
                

That single conformance_rate number is what you'd track in a dashboard or CI log over time. The failure_breakdown is what you'd actually read when something drops, since it tells you whether the model is dropping fields, hallucinating extra ones, or getting case-sensitive enums wrong, each of which usually points to a different fix (tightening the prompt, adding few-shot examples, or switching to a stricter decoding mode if your provider offers one).

If you're testing multiple prompts or multiple schema variants, wrap this in a small table so results are easy to scan at a glance:

                
def print_report(name, summary):
    print(f"{name:<20} {summary['conformance_rate']}% conformant "
          f"({summary['total_runs']} runs)")

print_report("extract_task", summary)
                

With this in place, you've turned an anecdotal "it seems to work most of the time" into a real, comparable metric. The next question is what to do with that number: what counts as passing, and what should trigger an alarm.

Reading the results: what counts as "good enough"?

A conformance rate on its own doesn't tell you whether to ship. 94% sounds fine until you think about what it means in practice: if this schema backs a workflow that runs a thousand times a day, that's 60 broken payloads hitting your system before lunch. The right threshold depends entirely on what happens downstream when validation fails.

A few rough guidelines:

• Production-critical pipelines (billing, data ingestion, anything writing to a database without a human in the loop) should be held to 99%+ conformance, and honestly, closer to 99.9% if failures are silent or expensive to unwind. If you can't hit that reliably, the fix isn't "test less," it's adding a retry-on-failure step or a stricter decoding mode before the output ever reaches production.
• User-facing but recoverable flows (a form autofill, a suggested tag, anything a person can review or correct) can tolerate more, maybe 90 to 95%, as long as failures degrade gracefully instead of crashing.
• Exploratory or internal tooling can often live with 80 to 90%, since the cost of an occasional bad output is low and a human is usually watching anyway.

These aren't hard rules, they're a starting point for a conversation your team should have explicitly, ideally before conformance testing turns up a number nobody agreed on in advance.

It's also worth testing conformance across a few dimensions instead of treating it as one fixed number for a prompt:

• Schema complexity. A flat, 3-field schema and a deeply nested one with optional arrays will not conform at the same rate, even with the identical model and prompt style.
• Input length and ambiguity. Short, clean inputs tend to conform more reliably than long or messy ones. If your real-world inputs are messy, test with messy inputs, not just the clean example in your prompt.
• Temperature. Lower temperature generally improves structural conformance, since the model has less room to improvise. If you're running temperature above 0 for creativity reasons, expect a lower conformance rate and test at the temperature you'll actually use in production.

The takeaway: don't treat conformance rate as a single pass or fail gate for "the model." Treat it as a property of a specific prompt, schema, and configuration combination, and set your threshold based on what breaks downstream if that combination fails.

Making this part of your regression suite

A conformance test you run once and forget is barely better than not testing at all. Models get updated behind the API you call, providers ship new versions, and prompts drift as other people on your team tweak them. The real value of this harness comes from running it regularly and catching the moment conformance drops, not from a one-time report.

The simplest way to do this is to fold the harness into your existing test suite as a scheduled job rather than a per-commit check, since running 50+ live model calls on every pull request gets slow and expensive fast. A nightly run is usually the right cadence.

                
# conformance_check.py
MIN_CONFORMANCE_RATE = 95.0

def run_regression_check():
    results = [run_once() for _ in range(50)]
    summary = summarize(results)

    print_report("extract_task", summary)

    if summary["conformance_rate"] < MIN_CONFORMANCE_RATE:
        raise SystemExit(
            f"Conformance dropped to {summary['conformance_rate']}%, "
            f"below threshold of {MIN_CONFORMANCE_RATE}%. "
            f"Failures: {summary['failure_breakdown']}"
        )

if __name__ == "__main__":
    run_regression_check()
                

Wire that into a scheduled CI job, and a non-zero exit code fails the build the same way any other broken test would:

                
# .github/workflows/conformance.yml
name: JSON Conformance Check
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: pip install jsonschema openai
      - run: python conformance_check.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
                

A few things worth building in once this is running regularly:

• Track the rate over time, not just pass or fail. A single build failure tells you something broke. A trend line tells you whether a provider is slowly getting worse, or whether a recent prompt change helped or hurt.
• Test across model versions explicitly. Providers don't always announce behavioral changes to structured output. If you're pinned to a model string, rerun this suite before accepting an upgrade, don't assume the new version is at least as conformant as the old one.
• Test across providers if you support more than one. Conformance rates for the same schema and prompt can vary noticeably between providers, and even between different models from the same provider.
• Alert on the failure breakdown, not just the rate. A drop from 98% to 96% because of one new failure type is often a more useful signal than the number itself.

Once this is in place, schema conformance stops being something you find out about from a downstream bug report and becomes something your test suite catches on its own.

Conclusion

Content correctness gets all the attention in LLM testing, but structural reliability is just as easy to break and far easier to miss, since a malformed field doesn't announce itself the way a wrong answer does. Schema conformance is measurable, it's cheap to test, and it behaves like any other SLA: define a threshold, monitor it, and get alerted when it slips.

The harness in this post is intentionally small. A prompt, a schema, a loop, and a validator are enough to turn "the model seems to work" into a number your team can actually trust and track over time. The complete, runnable code example is available on our GitHub page.