In this article
Before you start
Why interviewers ask this
A candidate who answers "I would ask it nicely to return JSON" has not met a malformed response in production. The topic appears in GenAI and prompting rounds as "how do you ensure the model outputs a fixed format", "extract fields from an invoice and handle malformed output", and "validate, then repair or retry". Reported in interviews at AI labs and large tech companies, it is often a coding exercise, listed among the prompt engineering interview questions: parse, validate against a schema and decide what to do on each kind of failure.
A weak answer relies on the prompt. A strong answer layers prevention and detection, and says what each layer cannot catch.
Getting reliable JSON from an LLM from first principles
In plain words
- Structured output
- A structured output is a model response that is promised or checked to follow a machine-readable format, usually a JSON document that matches a stated schema, so that code can consume it reliably (its fields and types are known in advance).
- JSON mode
- JSON mode is a provider setting that makes the model return syntactically valid JSON but does not make it follow any particular schema, so required fields and types can still be wrong.
- Constrained decoding
- Constrained decoding is a way of generating text in which, at each step, the tokens that would break a grammar or schema are removed before the next token is chosen, so every prefix stays valid, and the finished text fits the format whenever generation is allowed to finish.
- Validate-and-repair loop
- A validate-and-repair loop is a piece of application code that parses each model response, checks it against a schema, and on failure either fixes it mechanically or asks the model again, up to a fixed number of attempts.
Think of a form at a government office. You can tell a person to fill it in neatly, which sometimes works. You can give them a form with boxes that only accept digits in the date field, which prevents some errors. And a clerk still checks the finished form before stamping it.
The analogy breaks because the model can refuse to fill the form, or run out of paper halfway.
In one precise sentence, reliable JSON is a request that asks, a decoder that constrains and a validator that has the last word.
Step 1: Ask with a schema and an example
The cheapest layer is the prompt: state the exact fields, their types and an example. It helps but promises nothing, because the model still picks every token freely and may add prose around the JSON or write a number as a string.
A schema also costs context, as tool definitions do. The small invoice schema in the worked example is 209 characters long1 when written as compact JSON. Keep descriptions short and put range rules in the validator, for a reason Step 2 explains.
The input is a task and a schema in plain language. The output is a response that is usually right and sometimes wrapped in chatter.
Step 2: Provider modes: JSON mode, then schema-constrained output
Providers offer two levels, and they differ in what they promise. OpenAI's guide says both JSON mode and Structured Outputs ensure valid JSON, but only Structured Outputs ensures the output follows your schema; Structured Outputs is switched on with a JSON schema and a strict flag. Anthropic's documentation describes constraining the response to a schema you supply, through constrained decoding, with the compiled grammar cached after the first request.
Both come with limits worth knowing. The first request for a new schema has extra latency in both vendors' documentation. Only a subset of JSON Schema is supported: Anthropic lists recursive schemas, numeric limits such as minimum and maximum, and string-length limits as unsupported, and an unsupported feature returns an error. OpenAI notes a refusal does not follow your schema, and that a response cut off by a token limit may not parse. As of October 2026 these are the documented behaviors; check the current page for the model you use.
The input is a request with a schema and a strict setting. The output is a response that is schema-shaped except when refused or truncated.
Step 3: How the constraint works, and how to run it yourself
A schema-constrained provider mode is constrained decoding, so this step opens the box rather than adding a layer. Constrained decoding enforces the format inside the generation loop by masking illegal tokens. The idea in Willard and Louf (2023) is to recast generation as moves through a finite-state machine built from the grammar, with an index over the vocabulary, so the allowed tokens at each state can be found cheaply. Their open-source implementation is the library Outlines, for models you host.
A toy step makes it concrete. After the text {"age": only digits are legal. The model's raw scores for each token, called logit values, come first. With illustrative scores, the digit tokens "7" and "3" together hold only 0.1 of the probability, because the quote mark (opening a string) and "twenty" are more likely. Masking the illegal tokens and renormalizing gives "7" a probability of 0.6225 and "3" a probability of 0.37751. The cross-entropy loss page explains the score-to-probability step that this mask sits between.
Masking ensures syntax, not truth: the digits it allows may still be the wrong age.
The input is the model's scores for the next token and the grammar's current state. The output is a distribution over legal tokens only.
Step 4: Validate, classify and repair every response
Validation is the layer that cannot be skipped, because refusals, truncation and wrong values get past the constraints. The loop is: parse, validate against the schema, classify, then repair or retry. Six sample outputs for an invoice with a name, a date and an amount show the classes: a clean one is valid; one wrapped in the words "Sure! Here is the JSON" and one cut off mid-number fail to parse; a string where the amount should be a number fails on type; a date written 14/03/2026 fails the date pattern; and a missing amount fails on the required field1.
Each class has its own fix. Prose around JSON can be repaired mechanically by taking the text from the first opening brace to the last closing one, after which the example validates. A type or pattern failure goes back to the model with the validator's message. Truncation needs a higher token limit, and a refusal needs a human or a fallback, not a retry.
How many retries are worth it
Retries have diminishing returns. If each attempt is valid with probability 0.9, an illustrative assumption, three attempts give 0.999, and the expected number of calls with at most three attempts is 1.111. Failures are correlated in real systems, so cap the attempts and log the failures.
Step-by-step description
- The prompt helps but promises nothing.
- JSON mode gives syntax; structured outputs give the schema.
- Masking keeps only legal continuations; it cannot stop a refusal or a cut-off.
- Check refusals, truncation, types, then retry a few times.
- Masking gives the token 7 a probability of 0.6225 and the token 3 a probability of 0.3775.
- Six sample outputs fall into valid, unparseable and invalid classes.
- Three attempts at 0.9 validity per attempt give 0.999.
Knowing it is half the job. Saying it is the other half.
Interview Clozo is an AI interview copilot for real interview rounds: it hears the question and puts a strong answer on your screen, and you say it in your own words. See the features or how it works.
Start Free Trial →Worked example
| Sample | Class | What failed |
|---|---|---|
| Clean JSON | valid | nothing |
| Prose before the JSON | unparseable | text outside the object |
| Amount as a string | invalid | type |
| Date as 14/03/2026 | invalid | pattern |
| Missing amount | invalid | required field |
| Cut off mid-number | unparseable | truncation |
The schema requires a string name, a date matching four digits, two digits and two digits separated by hyphens, and a numeric amount, with no extra fields1.
Text description of the diagram
- Six sample outputs for an invoice fall into valid, unparseable and invalid classes.
- Prose around JSON can be repaired mechanically by taking the text from the first opening brace to the last closing one.
| Token | Before masking | After masking |
|---|---|---|
| " | 0.4906 | 0.0000 |
| twenty | 0.2789 | 0.0000 |
| 7 | 0.0622 | 0.6225 |
| 3 | 0.0377 | 0.3775 |
The scores are illustrative, not from a real model1.
Text description of the diagram
- Masking and renormalizing gives 7 a probability of 0.6225 and 3 a probability of 0.3775.
- Before masking the digit tokens 7 and 3 hold only 0.1 of the probability.
| Fact | Value |
|---|---|
| JSON mode ensures | valid JSON, not the schema |
| Structured Outputs ensures | valid JSON that follows the schema |
| First request with a new schema | extra latency |
| Refusal | may not match the schema |
| Cut-off response | may not parse |
| Attempts needed at 0.9 validity per attempt for 0.999 | 3 |
Trade-offs and edge cases
Each layer trades cost, coverage and flexibility, and the right stack depends on how bad a failure is. A short guide follows.
- Prompt only. Cheapest, with no promise. Acceptable for a prototype or a human-reviewed draft.
- JSON mode. Syntax only, so a validator is still required.
- Schema-constrained mode. Strongest, but limited to a schema subset, so range and length rules live in the validator. Check for refusals and truncation before trusting the result.
- Function calling. The same schema idea inside a tool call, covered in the tool use section of the Model Context Protocol page.
- Quality. Tam et al. (2024), in "Let Me Speak Freely?", report that format restrictions lower reasoning performance; the effect depends on the task, so measure it on a held-out set instead of assuming either way.
- Versioning. A schema is an interface, so version it and keep old fields working, and watch failure classes over time with LLM observability.
For high-stakes extraction, add a person to review the cases the validator marks unclear; see human-in-the-loop AI agents.
How to answer it in an interview
How do you make an LLM output reliable JSON, and what do you do when it does not?
I layer it. I describe the schema and an example in the request, then use the provider's schema-constrained mode, which is constrained decoding and masks illegal tokens during decoding so the syntax is valid. That still can't cover a refusal or a response cut off by the token limit, so I always parse and validate against a schema, using something like Pydantic. I classify failures: prose around JSON I extract mechanically, a wrong type or pattern goes back to the model with the error message, and I cap retries at two or three. Range checks live in the validator, not the schema.
The spoken structure is: ask, constrain, validate, classify the failure, then cap the retries.
Common mistakes
- "JSON mode ensures my schema" (wrong). It ensures valid JSON only; Structured Outputs adds the schema (right: validate anyway).
- "Constrained decoding means I can skip validation" (wrong). Refusals and truncation may not match the schema (right: check the stop reason first).
- "Retry until it works" (wrong). Unbounded retries cost money and hide a bad prompt or schema (right: cap attempts and log failures).
- "Put every rule in the schema" (wrong). Providers support only a subset, and numeric limits may be rejected (right: enforce ranges in your validator).
- "Parse with a regular expression" (wrong). Nested JSON defeats it (right: use a real JSON parser, then a schema validator).
Follow-up questions
How can we ensure the LLM outputs data in a fixed format, such as JSON or CSV?
Request the format with a schema, use a structured-output or constrained mode where one exists, and validate the result in code. For comma-separated values (CSV), constrain with a grammar or ask for JSON and convert, since a column count is easy to check after parsing.
What do you do when the model returns malformed JSON?
Classify the failure. Extract JSON from surrounding prose, raise the token limit after a cut-off, send type or format errors back to the model with the validator's message, and stop after a small number of attempts with a clear error.
What is the difference between JSON mode and structured outputs?
JSON mode returns valid JSON with no promise about its fields. Structured outputs constrain the response to your schema, so required fields and types match, apart from refusals and truncation.
Does constrained decoding hurt answer quality?
It can: Tam et al. (2024) report that format restrictions lower reasoning performance, and the effect depends on the task and the schema. Compare constrained and unconstrained outputs on a held-out set before deciding, and see the prompting hub for evaluation habits.
How do you handle a refusal or a truncated response?
Check the stop reason before parsing. A cut-off reply means raising the token limit or shortening the output; a refusal means no valid object exists, so route to a fallback or a person, as AI agents do for blocked steps.
- Layer the defenses: schema in the request, a structured mode (which is constrained decoding inside), then validation.
- JSON mode ensures syntax; schema-constrained modes ensure the schema, except for refusals and truncation.
- Constrained decoding masks illegal tokens and renormalizes, which fixes syntax and not truth.
- Classify failures: extract, repair with the error message, raise the limit, or escalate.
- Cap retries, keep range rules in the validator and log every failure class.
Sources
- Structured Outputs guide, OpenAI API documentation — OpenAI (page undated; read 2026-10-24), 24 Oct 2026
- Structured outputs, Claude API documentation — Anthropic (page undated; read 2026-10-24), 24 Oct 2026
- Efficient Guided Generation for Large Language Models (Willard and Louf), arXiv 2307.09702 v4 — arXiv (v1 submitted 2023-07-19), 19 Aug 2023
- Tool use with Claude (overview) — Anthropic (page undated; read 2026-10-24), 24 Oct 2026
- Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models (Tam et al.), arXiv 2408.02442 v3 — arXiv, 14 Oct 2024
