Structured outputs & JSON

6 min read · Updated 2026-08-10

If your code parses an LLM’s output, the output is an API — and an API that’s “usually valid JSON” is a bug waiting to happen. The good news: getting reliable structured output is a solved problem if you layer three things instead of trusting the prompt alone.

1. Ask for structure — and give the schema

Tell the model the exact shape you want, with types and which fields are required. A schema in the prompt does most of the work:

Return ONLY JSON, no prose, matching this schema:
{
  "priority": "low" | "medium" | "high",
  "summary": string,        // one sentence, <= 40 words
  "tags": string[]          // 0-5 short tags
}

Be explicit that the response must be JSON only — no greeting, no markdown code fence, no explanation. Prose around the JSON is the most common parse failure.

2. Use the provider’s structured-output mode

Most providers offer a JSON mode or structured outputs that constrains the decoder so the response is guaranteed to be syntactically valid JSON — and, with schema-constrained modes, guaranteed to match your schema. This is stronger than any prompt wording, because it operates on the generation itself. Use it whenever it’s available; keep the schema in the prompt too, since not every model supports constrained decoding.

3. Validate in code — never trust blindly

Always parse and validate the result against your schema (e.g., with a JSON Schema validator or a library like zod). Validation catches the cases the model gets subtly wrong — a missing field, a string where you expected a number, an enum value that isn’t in your list. On failure, you have three options: repair, retry, or fall back.

4. Handle truncation

A frequent failure isn’t malformed JSON — it’s incomplete JSON, because the model hit the output-token limit mid-object. Two defenses:

• Raise max_tokens to comfortably fit the largest expected output.
• Repair or retry when the JSON doesn’t close. A truncated object can often be repaired programmatically, or re-requested with a higher budget.

5. Keep the schema tight

Models produce better structured output when the schema is small and unambiguous:

• Prefer enums over free-form strings for categorical fields.
• Avoid deeply nested or optional-everywhere schemas.
• Give each field a one-line description of what it means.
• Don’t ask for fields the model can’t actually know — that’s where it hallucinates.

Putting it together

1. Prompt: "Return JSON only" + schema with enums and descriptions.
2. Call with JSON/structured-output mode enabled + generous max_tokens.
3. Parse → validate against the schema.
4. On failure → repair truncation, or retry once, or fall back.

That pipeline turns “usually JSON” into “always a valid object your code can trust.”

Checklist

• The prompt asks for JSON only and includes the schema.
• The provider’s JSON / structured-output mode is enabled.
• The response is validated against the schema in code.
• max_tokens is large enough to avoid truncation.
• There’s a repair/retry/fallback path for invalid output.