You ask an LLM to extract a JSON object from a support ticket. It returns something that looks like JSON, almost parses, and then your downstream code explodes on a missing comma. You tell the model "return valid JSON" in the system prompt, and it works 80% of the time, which is not enough for production.
Every major provider now ships a way to guarantee the model returns a schema-conformant object. They go by different names and use different mechanisms, but the goal is the same: stop parsing strings, start consuming typed data.
This article walks through the four approaches you will actually use in production: OpenAI's response_format, Anthropic's tool use with Claude 4, Gemini's responseSchema, and Ollama's format parameter for local models. Code is copy-paste, with the gotchas called out where they bite.
Why "please return JSON" does not work
A language model is a token predictor. It does not know what JSON is in the same way a parser does. The string "{" in its training data is followed by all sorts of things: object literals in JavaScript, math, prose. Asking nicely nudges the distribution but does not constrain it.
Common failure modes I have seen in real systems:
- Trailing commas in arrays or objects
- Markdown code fences wrapping the JSON (
json ...) - Comments like
// field below is optional - Truncated output when
max_tokensruns out mid-string - Mixed prose like "Here is the JSON: { ... } Hope this helps!"
- Wrong types: returning
"5"for an integer field, or"true"as a string
Structured output solves all of these by either constraining the sampling (Anthropic, OpenAI strict mode) or post-hoc validating the response against a schema (every provider supports this as a fallback).
Prerequisites
- Python 3.10+ with
pip install openai anthropic google-genai ollama pydantic - API keys for the cloud providers you want to test (or just the local Ollama path)
- For Ollama: install from ollama.com and pull a model that supports structured output (
ollama pull llama3.1:8b)
The shared pattern: Pydantic as the source of truth
Most Python projects settle on Pydantic for schema definition. It is expressive, has good validation, and most LLM SDKs have a path from Pydantic to the provider-specific schema format.
from pydantic import BaseModel, Field
from typing import Literal
class SupportTicket(BaseModel):
category: Literal["billing", "technical", "account", "other"]
priority: Literal["low", "medium", "high", "urgent"]
summary: str = Field(description="One-sentence summary of the issue")
needs_human: bool = Field(description="True if a human agent should follow up")
Each provider takes a different path from this model to a constraint. OpenAI builds a state machine. Anthropic leans on tool calls. Gemini eats the schema directly. Ollama hands it to the model's grammar.
OpenAI: json_schema with strict mode
OpenAI introduced structured outputs in August 2024. The strict mode uses constrained decoding: the API server builds a finite state machine from your schema and forces the model's output to traverse it token by token. The model literally cannot produce a string that does not parse.
from openai import OpenAI
import json
client = OpenAI()
resp = client.chat.completions.create(
model="gpt-4o-2024-08-06",
messages=[
{"role": "system", "content": "Extract structured data from support tickets."},
{"role": "user", "content": "My invoice for last month shows $250 but I was only billed $180. This is the second time. Please fix this and refund the difference."},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "support_ticket",
"strict": True,
"schema": SupportTicket.model_json_schema(),
},
},
)
ticket = SupportTicket.model_validate_json(resp.choices[0].message.content)
print(ticket.category, ticket.priority)
A few things to know:
- The
strict: trueflag is the magic. Without it, OpenAI falls back to a less reliable JSON-only mode that still occasionally produces bad output. additionalProperties: falseis required. Pydantic emits this by default inmodel_json_schema(), but if you hand-write a schema, add it.- All fields must be in
required. UseOptional[T] = Nonefor nullable fields; do not omit them from the schema. - The model is fixed to a snapshot that supports structured outputs. As of writing, the relevant IDs are
gpt-4o-2024-08-06and later,gpt-4o-mini-2024-07-18and later, plus the o1 family. Older snapshots ignore thejson_schematype.
Pydantic has a one-liner helper if you do not want to type out the wrapper:
pip install pydantic-ai
But for production code, calling the OpenAI client directly gives you more control over retries, timeouts, and streaming.
Anthropic: tool use as a structured output mechanism
Anthropic does not have a "response format" parameter in the same sense. Instead, you define a tool, mark it as required, and read the arguments. The model still has to call a tool, but the schema constraints make the arguments valid.
import anthropic
import json
client = anthropic.Anthropic()
tool = {
"name": "extract_ticket",
"description": "Extract structured fields from a support ticket",
"input_schema": SupportTicket.model_json_schema(),
}
resp = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
tools=[tool],
tool_choice={"type": "tool", "name": "extract_ticket"},
messages=[
{"role": "user", "content": "My invoice for last month shows $250 but I was only billed $180. This is the second time. Please fix this and refund the difference."},
],
)
# Find the tool use block
for block in resp.content:
if block.type == "tool_use" and block.name == "extract_ticket":
ticket = SupportTicket.model_validate(block.input)
print(ticket.category, ticket.priority)
break
Claude 4 also supports a feature called structured outputs (in beta) that uses constrained sampling like OpenAI. As of writing the beta header is structured-outputs-2025-09-15. If you are on the latest model and do not need streaming, prefer that over the tool-use trick, it has the same guarantees as OpenAI strict mode.
The tool-use pattern is still useful because:
- It works on every Claude model, including older ones that lack the beta feature
- It composes naturally with other tools in an agent loop
- The
tool_choiceparameter can force the model to call a specific tool, which is what you want for extraction
Google Gemini: responseSchema
Gemini takes a slightly different path. You pass a JSON Schema directly via generation_config.response_schema and set response_mime_type to application/json.
from google import genai
client = genai.Client()
resp = client.models.generate_content(
model="gemini-2.5-flash",
contents="My invoice for last month shows $250 but I was only billed $180. This is the second time. Please fix this and refund the difference.",
config={
"response_mime_type": "application/json",
"response_schema": SupportTicket.model_json_schema(),
},
)
ticket = SupportTicket.model_validate_json(resp.text)
print(ticket.category, ticket.priority)
Gemini supports constrained decoding on the 2.0 and later models. On older 1.5 models, the same parameter works but with weaker guarantees, so validate the response before trusting it.
A quirk worth knowing: Gemini does not handle Literal types the same way as OpenAI. It wants an enum in the JSON Schema, which Pydantic does emit correctly, but if you build the schema by hand, use "enum": [...] not "type": "literal".
Ollama: format parameter for local models
Local models are the weak link in this story. Not all of them support constrained decoding, and the ones that do may not support your specific schema type.
Ollama's format parameter accepts either a schema name ("json") or a full JSON Schema object. When you pass a schema, models that support grammar-constrained sampling (Llama 3.1, Qwen 2.5, Mistral Nemo) will be steered into producing valid output.
import ollama
resp = ollama.chat(
model="llama3.1:8b",
messages=[
{"role": "user", "content": "My invoice for last month shows $250 but I was only billed $180. This is the second time. Please fix this and refund the difference."},
],
format=SupportTicket.model_json_schema(),
)
ticket = SupportTicket.model_validate_json(resp["message"]["content"])
print(ticket.category, ticket.priority)
The gotcha: format with a JSON Schema only works on models that ship with a compatible GBNF grammar. The full list lives in the Ollama structured outputs docs. For unsupported models, the only option is format="json", which just nudges the model to return JSON without enforcing a schema. In that case, you must validate the response and retry on failure.
For models that support it, the output is constrained at the token level. For models that do not, you get a "best effort" JSON string that still needs parsing and validation.
Defensive parsing: validate everything, even constrained output
Schema-constrained output is not a substitute for validation. The model cannot produce malformed JSON, but the model can still produce a value that does not make sense for your domain (a priority of "super-urgent" in a free-text field, a summary that is empty, a category that is technically valid but semantically wrong).
Always run Pydantic validation on the response, even when the provider guarantees the schema:
from pydantic import ValidationError
def extract_ticket(text: str, max_retries: int = 2) -> SupportTicket:
for attempt in range(max_retries + 1):
try:
raw = call_provider(text) # your provider-specific function
return SupportTicket.model_validate_json(raw)
except ValidationError as e:
if attempt == max_retries:
raise
# Feed the error back to the model and retry
text = f"{text}\n\nPrevious attempt failed: {e}. Try again with valid values."
raise RuntimeError("unreachable")
For unconstrained paths (Ollama without grammar support, older Gemini models, or any provider in non-strict mode), wrap the call in a retry loop and feed validation errors back. The model usually fixes the issue on the second try because the error message tells it exactly what is wrong.
Performance and cost
Structured output is not free. Constrained decoding builds a state machine per request and the model has to traverse it, which adds latency and reduces throughput.
In my testing on gpt-4o-mini:
- Plain JSON prompt: ~0.6s median latency
- Strict
json_schema: ~0.9s median latency
That is a 50% slowdown for a guarantee that the output is valid. For high-throughput batch jobs, it is worth measuring whether the cost of retries on plain JSON (which average around 5-10% failure rates) is cheaper than the latency hit of strict mode.
For agent loops that call models in sequence, the latency is usually irrelevant. The retry path is the expensive one. Strict mode eliminates retries, which dominates.
When to use what
OpenAI strict mode if you are all-in on OpenAI. Best developer experience, fastest iteration, broadest model support.
Anthropic tool use if you are using Claude and want a pattern that works across all model versions. Switch to the structured outputs beta when you control the model version and do not need streaming.
Gemini responseSchema if Gemini is your primary provider or you need its pricing/throughput. The response_mime_type + response_schema pair is the cleanest API of the four.
Ollama format for local development, privacy-sensitive workloads, or cost-zero batch processing. Make sure you pick a model from the supported list; otherwise you are back to "please return JSON" and you need the retry loop.
For multi-provider setups (using OpenAI for production, Ollama for local dev), the cleanest abstraction is a small wrapper that converts a Pydantic model to each provider's format. Pydantic AI, Instructor, and Outlines all do this, and for a 2-3 provider setup, the integration is about 50 lines of code. For more providers, pick a library.
Common failure modes in production
Streaming with constrained output. OpenAI supports streaming structured outputs, but the response comes in deltas, not a complete object. You need a partial JSON parser to assemble it. The instructor library handles this; doing it by hand is a footgun.
Nested optionals and unions. Pydantic's Union[A, B] and Optional[A] produce JSON Schemas with oneOf and anyOf. All four providers support these, but the constraint engine is slower and some models occasionally fail to pick the right branch. Test edge cases explicitly.
Long strings. Schema-constrained output works fine for short text, but if you ask the model to generate a 10,000-token essay constrained to a schema, the sampling can get stuck. For long-form generation, stream without constraints and validate the final result.
Schema drift. When you change a Pydantic model, you change the contract. OpenAI caches the schema by name; if you change the schema without bumping the name, the old one might be served. Always bump the schema name on a breaking change, or use a hash.
Reference implementation
A small wrapper that works across all four providers:
from typing import Type, TypeVar
from pydantic import BaseModel
T = TypeVar("T", bound=BaseModel)
def extract(provider: str, model: str, text: str, schema: Type[T]) -> T:
if provider == "openai":
return _openai_extract(model, text, schema)
elif provider == "anthropic":
return _anthropic_extract(model, text, schema)
elif provider == "gemini":
return _gemini_extract(model, text, schema)
elif provider == "ollama":
return _ollama_extract(model, text, schema)
else:
raise ValueError(f"Unknown provider: {provider}")
Each _provider_extract function is 10-20 lines: build the provider-specific request, call the API, validate with Schema.model_validate_json (or Schema.model_validate for Anthropic's tool input), return the typed object. The total surface area is small enough that maintaining a custom wrapper beats fighting with a library that may or may not support your mix of providers and models.
The bigger architectural question is whether you want library-level abstraction at all. For a single-provider setup, you do not need it. For multi-provider, the trade-off is between writing 200 lines of glue code or accepting a library's opinionated error handling, retry policy, and streaming behavior. I lean toward the glue code for production systems where the failure modes need to be visible, and toward the library for prototypes where iteration speed matters.