Can you trust the JSON an LLM gives you if it matches your schema? No. Valid JSON with the right types can still carry the wrong values.
Say you ask a model for an integer called bedrooms, and the listing text mentions a built-up area of 1,450 square feet. The model may hand you "bedrooms": 1450.
The JSON parses, the type is right, the shape matches. You asked for the right shape and got it; you did not get the right answer.
This post shows how I close that gap:
- One Zod schema at the boundary.
- One retry with the error fed back.
- A hard cap of two attempts.
- A per-field rule for when to fail loudly.
Prompt schemas, JSON mode and tool calls promise different things
There are three ways to ask a large language model for JSON, and they promise different things.
- A schema in the prompt ("respond only with JSON") promises nothing. You get a code fence sometimes, a sentence before it when the input is unusual, and a trailing comma once a month.
- Provider JSON modes and structured outputs promise syntax. The response parses, and with a JSON Schema attached, the keys and types match it. I turn that on everywhere it exists.
- Tool-call mode promises the same thing with a different framing: you define a function with typed parameters, and the arguments arrive as a parsed object. For extraction I prefer this, because a tool definition reads as a contract and the model treats it like one.
None of the three promise the values are right.
A status typed as an enum of four strings will contain one of the four; whether it is the correct one is the model's judgement. A publishedAt typed as string will be a string. It may be "last Tuesday".
So treat the response the way you would treat a form post from the internet: useful, probably fine, not trusted.
Validate LLM output with one Zod schema
One schema, defined once, in Zod:
import { z } from "zod";
export const Listing = z.object({
title: z.string().min(3).max(200),
price: z.number().int().positive(),
currency: z.enum(["INR", "USD"]),
bedrooms: z.number().int().min(0).max(20).nullable(),
furnished: z.enum(["full", "semi", "none"]),
publishedAt: z.string().datetime({ offset: true }),
summary: z.string().max(400),
});
export type Listing = z.infer<typeof Listing>;The max(20) on bedrooms is not there because I expect a 21-bedroom flat. It is there because 1,450 is a number the model will offer again, and the schema is the place to say no.
The type comes from the schema, never a hand-written interface beside it, so the two cannot drift.
The same schema generates the JSON Schema that goes in the request:
import { zodToJsonSchema } from "zod-to-json-schema";
const tool = {
name: "record_listing",
description: "Record the structured fields of a property listing. bedrooms is a count of rooms, never an area.",
input_schema: zodToJsonSchema(Listing),
};That sentence in the description, "never an area", stops most wrong answers. Validation catches the rest:
const raw = extractToolInput(response, "record_listing");
const parsed = Listing.safeParse(raw);
if (!parsed.success) {
throw new ModelOutputError(parsed.error, raw);
}
return parsed.data; // Listing, and nothing downstream sees anything elsesafeParse rather than parse, because I want the failure as a value I can log and send back to the model. An exception loses the raw output on its way up the stack.
Retry once with the error, then stop at two
When validation fails, the model can usually fix it if you tell it what was wrong.
async function extract(input: string, attempt = 1, lastError?: string): Promise<Listing> {
const response = await callModel(buildMessages(input, lastError));
const parsed = Listing.safeParse(extractToolInput(response, "record_listing"));
if (parsed.success) return parsed.data;
if (attempt >= MAX_ATTEMPTS) throw new ModelOutputError(parsed.error, response, attempt);
// The error travels with the call, never in module state shared between requests.
const issues = formatZodIssues(parsed.error); // "bedrooms: must be <= 20, got 1450"
return extract(input, attempt + 1, issues);
}- 1Call the modeltool call with the Zod-derived schema
- 2safeParseshape and values checked at the boundary
- 3Retry oncewith the validation error in the prompt
- 4Stop at 2 attemptseach one is a full billed call
- 5Fail loudlylog raw output and Zod issues together
MAX_ATTEMPTS is 2. A model that has failed the same schema twice on the same input is usually failing because the input does not contain what the schema wants. No number of retries puts it there.
Here's the catch: each retry is a full call at full price, with input tokens, schema and system prompt billed again.
If validation fails five percent of the time and you retry once, your cost per answer is up five percent before you have shipped anything. I worked through that arithmetic for a different feature in what a RAG answer costs.
Log the retry rate per field. When one field dominates, the fix is in the prompt or the schema, and retrying harder will not find it.
If calls are timing out rather than failing validation, that is a different loop, covered in LLM API timeouts, retries and fallbacks.
Decide per field: default or fail loudly
This is the part I think most code gets wrong, and it gets it wrong in the quiet direction.
The temptation is to default a field that failed. Store null for bedrooms, move on.
The request succeeds, nobody is paged, and it feels robust in the way a smoke detector with the battery out is quiet.
With a silent default, the bedrooms bug never gets a ticket. A listing with no bedroom count is common enough that nobody reports it. The field is null on every listing whose description leads with its area, and the first person to notice is whoever runs the quarterly report.
My rule is per field, decided when the schema is written:
- Fields that may default are the ones a human will shrug at if they are missing, like the summary or a tag list. Mark them
.optional()or.catch(fallback)so the decision is visible in code. - Fields that must fail are price, currency, the enum that drives business logic, and anything code downstream will trust. A failure after retries is a failed request with the raw output logged.
If I had to pick one side for a whole feature, I would pick loud.
A feature that fails visibly one time in two hundred gets fixed. A feature that corrupts one record in two hundred gets discovered a quarter later, with four hundred bad rows and no raw outputs to explain them.
The open problem: drift between model versions
Everything above treats the model as untrusted input, the same stance I take in the production RAG chatbot.
What I do not have a good answer for yet is drift. A model version changes, and the retry rate on one field goes from zero to three percent.
Nothing is broken, exactly:
- The bill is a little higher.
- The latency has a longer tail.
- No alert fires, because no request failed.
I am leaning towards treating validation-failure rate per field as a metric with a threshold, like error rate. I have not run that long enough to know whether the thresholds are tunable or just noise.
Conclusion
Validate every LLM response against one Zod schema at the boundary, retry once with the error, and fail loudly on the fields that matter. Structured output gets you valid JSON; only validation gets you values you can trust.
If your extraction feeds a retrieval feature, the next read is building a production RAG chatbot, which applies the same untrusted-input stance end to end.
Which field in your schema would do the most damage if it were quietly wrong?