Skip to content
Sandeep Kumar
Updated 6 min read

Getting JSON you can trust out of an LLM

A schema in the prompt is a request, not a guarantee. How one bad field slips through, and how I validate LLM JSON output with Zod, retry once, and fail loudly.

Sandeep Kumar, Full-stack EngineerSandeep KumarFull-stack Engineer
Getting JSON you can trust out of an LLM: One extraction, from call to a result you can trust
On this page
  1. Prompt schemas, JSON mode and tool calls promise different things
  2. Validate LLM output with one Zod schema
  3. Retry once with the error, then stop at two
  4. Decide per field: default or fail loudly
  5. The open problem: drift between model versions
  6. Conclusion

Can you trust the JSON an LLM gives you if it matches your schema? No. Valid JSON with the right types can still carry the wrong values.

Say you ask a model for an integer called bedrooms, and the listing text mentions a built-up area of 1,450 square feet. The model may hand you "bedrooms": 1450.

The JSON parses, the type is right, the shape matches. You asked for the right shape and got it; you did not get the right answer.

This post shows how I close that gap:

  • One Zod schema at the boundary.
  • One retry with the error fed back.
  • A hard cap of two attempts.
  • A per-field rule for when to fail loudly.

Prompt schemas, JSON mode and tool calls promise different things

There are three ways to ask a large language model for JSON, and they promise different things.

  • A schema in the prompt ("respond only with JSON") promises nothing. You get a code fence sometimes, a sentence before it when the input is unusual, and a trailing comma once a month.
  • Provider JSON modes and structured outputs promise syntax. The response parses, and with a JSON Schema attached, the keys and types match it. I turn that on everywhere it exists.
  • Tool-call mode promises the same thing with a different framing: you define a function with typed parameters, and the arguments arrive as a parsed object. For extraction I prefer this, because a tool definition reads as a contract and the model treats it like one.

None of the three promise the values are right.

A status typed as an enum of four strings will contain one of the four; whether it is the correct one is the model's judgement. A publishedAt typed as string will be a string. It may be "last Tuesday".

So treat the response the way you would treat a form post from the internet: useful, probably fine, not trusted.

Validate LLM output with one Zod schema

One schema, defined once, in Zod:

ts
import { z } from "zod";

export const Listing = z.object({
  title: z.string().min(3).max(200),
  price: z.number().int().positive(),
  currency: z.enum(["INR", "USD"]),
  bedrooms: z.number().int().min(0).max(20).nullable(),
  furnished: z.enum(["full", "semi", "none"]),
  publishedAt: z.string().datetime({ offset: true }),
  summary: z.string().max(400),
});

export type Listing = z.infer<typeof Listing>;

The max(20) on bedrooms is not there because I expect a 21-bedroom flat. It is there because 1,450 is a number the model will offer again, and the schema is the place to say no.

The type comes from the schema, never a hand-written interface beside it, so the two cannot drift.

The same schema generates the JSON Schema that goes in the request:

ts
import { zodToJsonSchema } from "zod-to-json-schema";

const tool = {
  name: "record_listing",
  description: "Record the structured fields of a property listing. bedrooms is a count of rooms, never an area.",
  input_schema: zodToJsonSchema(Listing),
};

That sentence in the description, "never an area", stops most wrong answers. Validation catches the rest:

ts
const raw = extractToolInput(response, "record_listing");
const parsed = Listing.safeParse(raw);

if (!parsed.success) {
  throw new ModelOutputError(parsed.error, raw);
}

return parsed.data;   // Listing, and nothing downstream sees anything else

safeParse rather than parse, because I want the failure as a value I can log and send back to the model. An exception loses the raw output on its way up the stack.

Retry once with the error, then stop at two

When validation fails, the model can usually fix it if you tell it what was wrong.

ts
async function extract(input: string, attempt = 1, lastError?: string): Promise<Listing> {
  const response = await callModel(buildMessages(input, lastError));
  const parsed = Listing.safeParse(extractToolInput(response, "record_listing"));

  if (parsed.success) return parsed.data;
  if (attempt >= MAX_ATTEMPTS) throw new ModelOutputError(parsed.error, response, attempt);

  // The error travels with the call, never in module state shared between requests.
  const issues = formatZodIssues(parsed.error);   // "bedrooms: must be <= 20, got 1450"
  return extract(input, attempt + 1, issues);
}
  1. 1Call the modeltool call with the Zod-derived schema
  2. 2safeParseshape and values checked at the boundary
  3. 3Retry oncewith the validation error in the prompt
  4. 4Stop at 2 attemptseach one is a full billed call
  5. 5Fail loudlylog raw output and Zod issues together
One extraction, from call to a result you can trust

MAX_ATTEMPTS is 2. A model that has failed the same schema twice on the same input is usually failing because the input does not contain what the schema wants. No number of retries puts it there.

Here's the catch: each retry is a full call at full price, with input tokens, schema and system prompt billed again.

If validation fails five percent of the time and you retry once, your cost per answer is up five percent before you have shipped anything. I worked through that arithmetic for a different feature in what a RAG answer costs.

Log the retry rate per field. When one field dominates, the fix is in the prompt or the schema, and retrying harder will not find it.

If calls are timing out rather than failing validation, that is a different loop, covered in LLM API timeouts, retries and fallbacks.

Decide per field: default or fail loudly

This is the part I think most code gets wrong, and it gets it wrong in the quiet direction.

The temptation is to default a field that failed. Store null for bedrooms, move on.

The request succeeds, nobody is paged, and it feels robust in the way a smoke detector with the battery out is quiet.

With a silent default, the bedrooms bug never gets a ticket. A listing with no bedroom count is common enough that nobody reports it. The field is null on every listing whose description leads with its area, and the first person to notice is whoever runs the quarterly report.

My rule is per field, decided when the schema is written:

  • Fields that may default are the ones a human will shrug at if they are missing, like the summary or a tag list. Mark them .optional() or .catch(fallback) so the decision is visible in code.
  • Fields that must fail are price, currency, the enum that drives business logic, and anything code downstream will trust. A failure after retries is a failed request with the raw output logged.

If I had to pick one side for a whole feature, I would pick loud.

A feature that fails visibly one time in two hundred gets fixed. A feature that corrupts one record in two hundred gets discovered a quarter later, with four hundred bad rows and no raw outputs to explain them.

The open problem: drift between model versions

Everything above treats the model as untrusted input, the same stance I take in the production RAG chatbot.

What I do not have a good answer for yet is drift. A model version changes, and the retry rate on one field goes from zero to three percent.

Nothing is broken, exactly:

  • The bill is a little higher.
  • The latency has a longer tail.
  • No alert fires, because no request failed.

I am leaning towards treating validation-failure rate per field as a metric with a threshold, like error rate. I have not run that long enough to know whether the thresholds are tunable or just noise.

Conclusion

Validate every LLM response against one Zod schema at the boundary, retry once with the error, and fail loudly on the fields that matter. Structured output gets you valid JSON; only validation gets you values you can trust.

If your extraction feeds a retrieval feature, the next read is building a production RAG chatbot, which applies the same untrusted-input stance end to end.

Which field in your schema would do the most damage if it were quietly wrong?

FAQs

Does JSON mode or structured output make validation unnecessary?

No. Those modes make the output parse as JSON and, with a schema attached, match its shape. They do not stop the model putting a plausible wrong value in a correctly typed field, a date in the wrong format, or an enum value it invented. Validation at the boundary catches the second class of error, which is the one that reaches users.

How many times should I retry a failed LLM response?

Once with the validation error included in the prompt, and at most twice in total. Each retry is a full call at full price, and a model that failed twice on the same input is usually failing for a reason a third attempt will not fix. After the cap, fail the request and log the raw output.

Should I use Zod or JSON Schema for LLM output?

Use Zod in TypeScript code and generate JSON Schema from it for the request. One definition then drives the prompt, the runtime check and the static type. Writing the schema twice is how the two versions end up disagreeing six weeks later.

What should happen when LLM output fails validation after retries?

The request should fail, visibly, with the raw model output and the validation errors logged together. Quietly substituting a default for a field that drives business logic turns a one-in-two-hundred model error into a slow data-corruption problem that nobody notices until a report does not reconcile.

Found this useful?Share on XShare on LinkedIn

Related reading