Beyond Structured Outputs: The Validation Layer LLMs Still Need
Structured outputs (constrained decoding) guarantee that an LLM’s response is valid against your JSON Schema. They do not guarantee that the response is correct, because providers implement only the subset of JSON Schema that compiles to a grammar: no string length limits, no numeric bounds, no minItems above 1 and no cross-field rules. The grammar owns shape; a validator in your own code still owns meaning. We deleted a third of our parsing code when structured outputs arrived and kept the semantic validator, because it is the only thing that stops a well-formed wrong answer from reaching a user.
Quick answer
Does structured output remove the need for validation? No. It removes the need for a tolerant parser (fence stripping, key aliasing, type coercion). It does not check counts, lengths, uniqueness or agreement between fields. Keep a validator for every rule that is about meaning.
Definitions
Structured outputs (constrained decoding): a provider feature that compiles your JSON Schema into a grammar and, at each decoding step, masks out any token that would leave a valid path. Malformed JSON becomes unreachable.
Shape: what a schema language can express and a grammar can enforce: keys, types, enums, required fields, closed objects.
Meaning: what the output has to satisfy to be usable: counts, lengths, distinctness, cross-field agreement, domain rules. These live in code.
The short version
- A prompt is a request; a grammar is a constraint. Constrained decoding makes malformed JSON impossible, and that is all it does.
- Providers implement a subset of JSON Schema Draft 2020-12. On ours, minLength, maxLength, minimum, maximum, multipleOf, minItems above 1, recursive schemas and external $ref are unsupported.
- Every rule that made our content publishable lived in the unsupported half, so the validator stayed.
- Test with output that is valid but refusable; a malformed fixture only proves the parser, a well-shaped wrong fixture proves the validator.
The problem: asking is not the same as enforcing
Every team that generates content with an LLM writes the same defensive code twice. First a prompt that ends in “respond with JSON in this exact shape.” Then, a week later, the parser that copes with what comes back: a stray code fence, a preamble sentence, a key named “answer” instead of “correct_answer”, a number arriving as a string.
With a prompt, the model samples freely and hopefully lands on valid JSON; your parser copes when it does not. With constrained decoding, the provider compiles your schema into a grammar and masks out any token that would take the output off a valid path. Constrained decoding removes the box you wrote by hand. It does not remove the box below it. The temptation is to read “schema-valid” as “valid.” Those are different claims, and the second is the one the product needs.
Constrained decoding removes the box you wrote by hand.
It does not remove the box below it.
The temptation is to read "the output is now schema-valid" as "the output is
now valid." Those are different claims, and the second one is the one your
product actually needs.
The trap: a schema language is a subset
Providers do not implement all of JSON Schema. They implement the part that compiles into a deterministic grammar. Put real content rules next to that list and the gap is obvious.
Read the table top to bottom and the split names itself: shape goes in the schema, meaning stays in code. Anything counted, measured, compared across fields or checked for distinctness is meaning.
The grammar owns SHAPE. The validator owns MEANING.
The bottom row owns itself.
Read the table top to bottom and the split names itself: shape goes in the
schema, meaning stays in code. Anything counted, measured, compared across
fields, or checked for distinctness is meaning.
The approach: delete the parser, keep the judge
That split told us what to remove: fence stripping, brace hunting, key-alias guessing, string-to-int coercion and the default filling that quietly turned a half answer into a whole one. Roughly a third of the file, and the third that was hardest to reason about, because “tolerant” and “wrong” are neighbors.
The schema we send is deliberately boring, shape only:
{
"type": "object",
"additionalProperties": False,
"required": ["question", "options", "correct_answer", "explanation"],
"properties": {
"question": {"type": "string"},
"options": {"type": "array", "items": {"type": "string"}},
"correct_answer": {"type": "string"},
"explanation": {"type": "string"},
},
}
The validator behind it stayed intact, now the only thing between a generation and a published question:
def unpublishable(fields) -> bool:
options = fields["options"]
if not MIN_OPTIONS <= len(options) <= MAX_OPTIONS: # minItems can't say this
return True
if len(set(options)) != len(options): # nor uniqueness
return True
if any(len(o) > MAX_OPTION_CHARS for o in options): # nor maxLength
return True
if len(fields["explanation"]) < MIN_EXPLANATION:
return True
return fields["correct_answer"] not in options # nor cross-field agreement
Three operational details that are not in the happy path
- An unsupported keyword is a 400 at request time, not a relaxation. That is the good failure: a schema written from memory breaks the call loudly instead of silently accepting a 12-option question. It also means the provider’s keyword list is a build-time dependency.
- The first request for a new schema compiles a grammar, and compilation can take minutes. Compiled grammars are cached for about a day. Our generation is asynchronous and never sits on a page load, so a cold schema is invisible; on a synchronous read path the same fact is a latency incident.
- Feature interactions are real. On our provider, constrained decoding is mutually exclusive with citations and is rejected on one of the API surfaces. Check the compatibility matrix, not only the announcement.
The result, and one stub we deliberately kept useless
Malformed-JSON failures went to zero, and refusals now arrive with a named reason from the validator instead of a shrug from the parser.
The unexpected part came from the offline path. When no model is configured (local development, test runs without live credentials) calls fall through to a stub. Before structured outputs, that stub emitted an echo that happened to be unparseable as a question. Nobody designed that, but it had become load-bearing: a development run physically could not seed real content with filler.
Handing that stub a schema would have destroyed the property; it would have emitted well-formed, publishable-looking content into a real bank. So the stub is now schema-shaped on purpose and refusable on purpose. It walks the schema, knows nothing about the domain, and fills every string with the same placeholder, so every array of strings comes out as duplicates, and a duplicated choice is exactly what a multiple-choice validator must reject. The offline path exercises the entire pipeline and still cannot publish a single row. Filler in a scored bank is worse than an empty slot.
Where this validator sits in the wider pipeline, and how the prompt that feeds it is assembled, is covered in dynamic prompt construction at runtime. The regression suite that catches valid-but-wrong output is described in golden journeys: synthetic conversations as regression tests, and the routing layer that picks which model answers is in policy-driven model routing.
An unusable generation publishes nothing.
Filler in a scored bank is worse than an empty slot.
A well-formed answer just became the cheap part. Deciding whether it's a GOOD
answer is still entirely yours.
Takeaways
“Schema-valid” and “valid” are different claims; before planning a validation layer away, read which keywords the platform implements. Split on shape versus meaning: keys, types, enums and closed objects go to the grammar; counts, lengths, distinctness and cross-field agreement stay in code. Cross-field rules are never schema rules; if your domain has one, you have a validator permanently. Test with output that is valid but refusable. Keep constrained calls off synchronous paths until the grammar is warm.
Where this came from
This plumbing sits under a Hoomanely feature that generates daily care questions written for one specific animal (its breed, age, life stage and declared conditions) and then scores the owner’s answer. That last detail is why the validator is a product-safety layer rather than developer hygiene. In general contexts a reader can often tell when a model is wrong; in health contexts they usually cannot, and a confidently well-formed wrong answer is more dangerous than a malformed one. The same honesty applies to EverBowl™, which learns how a dog eats and drinks and does not diagnose anything; the value is the record, and the record has to be right. More LLM systems posts are in the LLM category.
Conclusion
Constrained decoding made our output well-formed. Keeping the validator is what keeps it answerable for.
FAQs
What do structured outputs guarantee?
That the response conforms to the subset of your JSON Schema the provider can compile into a grammar: keys, types, enums, required fields and closed objects. Nothing about lengths, counts, uniqueness or agreement between fields.
Which JSON Schema keywords are not supported by constrained decoding?
On our provider: minLength, maxLength, minimum, maximum, multipleOf, minItems above 1, recursive schemas, external $ref, and additionalProperties set to anything but false. Check your provider’s list; it is a build-time dependency.
Do I still need a validator if I use structured outputs?
Yes, for every rule about meaning: option counts, string lengths, duplicates, and any rule that compares one field with another. The grammar replaces your tolerant parser, not your judge.
What happens if my schema uses an unsupported keyword?
The request fails with a 400 at call time rather than being relaxed silently. That is the useful failure mode: it surfaces a schema written from memory before it accepts bad content.
Why can a cold schema cause latency?
The first request for a new schema compiles a grammar, which can take minutes, and the compiled grammar is cached for about a day. Keep constrained calls on asynchronous paths or warm the schema before it is needed on a page load.
How should I test a validation layer behind structured outputs?
With fixtures that are perfectly shaped and semantically wrong: duplicate options, an over-long option, a correct answer that is not in the list. A malformed fixture only proves the parser, which the grammar has already made redundant.
