Structured Output

Four Ways to Get JSON: Not the Same Order of Reliability

Background & principle

"Make the model output JSON" looks like one thing, but the underlying mechanisms differ wildly, from weak to strong:

• Prompt + prefill: write "return JSON only" in the system, then prefill a { to skip pleasantries. Zero guarantee — the model can still add fences, comments, or misspell fields.
• JSON mode: the API guarantees valid JSON, but not that it matches your schema. Parseable ≠ correct fields.
• Tool use / function calling: describe structure via the tool's input_schema and the model fills params. Stronger than JSON mode, but still not 100% in practice — OpenAI's own number for function calling is about 86% schema compliance.
• Constrained decoding (Structured Outputs): compile the schema into a grammar at the decoding layer and, at each step, only allow valid tokens to be sampled. Both OpenAI and Anthropic now offer a strict mode (Anthropic has opened Structured Outputs for Claude Sonnet 4.5 / Opus 4.1), claiming 100% schema compliance and zero parse errors.

The key engineering intuition: the first two are "pray afterwards," the last two are "constrain beforehand." Constrained decoding works by converting the JSON Schema into a CFG/FSM and masking every illegal token at each generation step — so it is physically impossible to emit a structure-violating token. That's also why it's cheaper than retry-until-valid: instead of generating then validating, you never let the wrong token be born.

Hands-on

Same extraction task, using Anthropic strict structured output (decode-layer guarantee) rather than prompt-for-JSON:

import anthropic
client = anthropic.Anthropic()

schema = {
  "type":"object",
  "properties":{
    "sentiment":{"type":"string","enum":["pos","neg","neutral"]},
    "key_entities":{"type":"array","items":{"type":"string"}}
  },
  "required":["sentiment","key_entities"],
  "additionalProperties":False
}
r = client.messages.create(
  model="claude-sonnet-4-5", max_tokens=1024,
  messages=[{"role":"user","content": review_text}],
  output_format={"type":"json_schema","schema": schema}  # decode-layer constraint
)
data = r.content[0].input  # already a dict, no json.loads + try/except

Note additionalProperties:false — strict mode usually requires it, or the model invents extra fields.

The Hidden Tax of Constrained Decoding: Reason First, Then Constrain

Background & principle

The 2024 EMNLP paper Let Me Speak Freely? (Tam et al.) dropped a bomb: across multiple reasoning benchmarks, forcing models to answer in JSON/XML formats significantly degraded reasoning accuracy, with stricter constraints degrading more. The intuitive explanation: constrained decoding masks out the tokens the model would have used to "think out loud," effectively stripping its chain-of-thought.

But the story isn't over. The Outlines team (.txt) published a rebuttal, Say What You Mean: on reproduction, they found that as long as the schema is well-designed and good few-shot structure examples are provided, structured generation doesn't drop scores — it can slightly raise them. The two aren't actually contradictory; together they yield one shippable conclusion: it's not structure that hurts, it's squeezing out the reasoning field that hurts.

So the real tactic is field-ordering engineering: JSON is generated autoregressively, so when the model writes later fields it can see the earlier ones it already wrote. Put the reasoning field before the answer field and the model does its CoT inside the structure, then produces the answer; put answer first and you force it to answer before thinking. When you need to go all the way, use two stages: stage one is free CoT with no constraint, stage two is the light task of "extract the conclusion into a schema."

Hands-on

# ❌ answer first: model is forced to answer before reasoning
{"answer": ..., "reasoning": ...}

# ✅ reasoning first: CoT embedded in structure, answer follows it
schema = {"type":"object",
  "properties":{
    "reasoning":{"type":"string",
      "description":"Think step by step before answering."},
    "answer":{"type":"number"}
  },
  "required":["reasoning","answer"]}  # required order = generation order

# ✅✅ reasoning-heavy task: two stages, CoT stage unconstrained
cot = ask(prompt)                       # free text, full reasoning
out = ask(f"Extract JSON from this analysis:\n{cot}", schema=schema)

Schema Is a Prompt: Field Design Decides Fill Quality

Background & principle

Constrained decoding guarantees "structurally legal," but whether the filled-in content is correct still depends on how the schema is written. This shares a root with Day 4's tool-use conclusion: description matters more than field name. A few repeatedly-validated design disciplines:

• Write a description for every field: the model treats it as a field-level instruction. "due_date" plus "ISO 8601, null if absent" is far more reliable than a bare field name.
• Use enum to converge categories: make classification fields enums, not free strings — constrained decoding physically guarantees a value within the set, stronger than "please pick one of the following."
• Flat beats deeply nested: nesting beyond three levels noticeably lowers fill accuracy and slows or even breaks some constraint engines. Flatten what you can.
• Be wary of optional / union: in strict mode OpenAI requires all fields in required, expressing optionality via a "type union null"; heavy anyOf unions have limited support in most constraint engines. "Explicitly fill null when missing" is more stable than a pile of optional fields.
• Leave room for injection defense: when extracting from untrusted text, don't let the model stuff raw text into fields that decide control flow — that's the Day 24 prompt-injection boundary.

Hands-on

A flat schema with descriptions and enums via Pydantic (both OpenAI/Anthropic SDKs eat it directly):

from pydantic import BaseModel, Field
from enum import Enum

class Priority(str, Enum):
    low="low"; med="med"; high="high"

class Ticket(BaseModel):
    """Extract a support ticket from a user message."""
    summary: str = Field(description="One-line problem, <=80 chars")
    priority: Priority = Field(description="high only if blocking/data-loss")
    due_date: str | None = Field(description="ISO 8601 or null")
    # flat + per-field description + enum + explicit null

This schema does three jobs at once: constrain the structure, use descriptions as field-level instructions, and use the enum to physically pin priority to three values. The reader isn't just the parser — it's the model itself.

The Fault-Tolerance Stack: When "100% Schema" Still Fails

Background & principle

"100% schema compliance" is an easily-misread promise. It guarantees the emitted tokens satisfy the grammar, but it can't handle any of these:

• Truncation: you hit max_tokens, stop_reason == "max_tokens", and what you got is structurally unclosed output. Always check stop_reason first, don't blindly parse.
• Refusal: the model may take a refusal branch, in which case the structured channel returns something other than your schema. Handle this path separately.
• Semantic errors: the schema says email: string, the model fills "none" — a legal string, but not an email. Structure right, semantics wrong.
• Streaming partials: mid-stream JSON is incomplete and needs an incremental/tolerant parser; you can't wait until the end to parse.

So the real production shape isn't "call once, get a dict," it's a validate-repair-fallback chain — the same resilience thinking as Day 3's harness "feed the error back to the model": handing a validation error back to the model to self-repair beats throwing an exception.

Hands-on

from pydantic import ValidationError

def extract(text, max_repair=2):
    msgs = [{"role":"user","content": text}]
    for _ in range(max_repair+1):
        r = call(msgs, schema=Ticket)
        if r.stop_reason == "max_tokens":      # 1) truncation: don't parse
            raise Truncated("raise max_tokens / split task")
        try:
            return Ticket.model_validate(r.data)  # 2) semantic validation
        except ValidationError as e:
            msgs += [{"role":"assistant","content": str(r.data)},
                     {"role":"user",
                      "content": f"Validation failed, fix and resend: {e}"}]  # 3) feed back
    raise Unrepairable()                          # 4) fallback: human/default

Four layers backstop in order: truncation detection → semantic validation (Pydantic validators cover emails, ranges, cross-field constraints the schema can't express) → feed the error back for self-repair → fall back if unfixable. Constrained decoding only covers the "structure" layer; the other three you build yourself.