You ask an LLM for JSON. What comes back is valid JSON wrapped in markdown fences, a sentence of preamble, and maybe a key you never asked for. Your parse fails, you write a regex, the regex breaks on the next edge case, and somewhere in your agent loop the code reading data["result"] chokes on a string that is actually None. Everyone building agents hits this wall.
The fix is not better prompt wording alone. It is structured outputs: passing a JSON schema to the model so token generation is constrained to match it. This article shows the practical version, using Pydantic for both the hosted OpenAI API and a local Ollama server. One schema definition, two backends, zero hand-rolled parsing.
The problem with just asking for JSON
When you only write "return JSON" in the prompt, the model uses JSON mode at best: valid JSON with no schema guarantee. It can rename keys, drop required fields, or return an object where you expected an array. JSON mode only promises the output is parseable. Structured outputs promise it matches your schema, key for key, type for type.
Structured Outputs (OpenAI) is the evolution of JSON mode. It compiles your JSON Schema into the decoder and masks any token that would produce a non-conforming output. Ollama does the same with constrained decoding when you pass a schema to the format parameter. The model physically cannot emit a markdown fence or a stray sentence, because that token would violate the grammar.
Prerequisites
- Python 3.10+ in a virtual environment
pip install openai pydanticfor the hosted APIpip install ollama pydanticfor the local server- Docker or a working Ollama install if you go local (see references)
Step 1: define the schema once with Pydantic
The one trick that makes both backends behave is defining your contract as a Pydantic model. model_json_schema() turns it into JSON Schema your provider accepts, and model_validate_json() validates whatever comes back.
from typing import Literal
from pydantic import BaseModel, Field
class CalendarEvent(BaseModel):
name: str = Field(description="Event name, e.g. 'Science fair'")
date: str = Field(description="ISO 8601 date")
participants: list[str] = Field(description="List of people going")
kind: Literal["meeting", "event", "task"] = "event"
The descriptions are not decoration. They travel inside the schema the model reads, so they work as instructions. Literal narrows the value to one of the allowed options. This same Pydantic model is used for validation, so there is a single source of truth.
Step 2: structured output from the OpenAI API
The OpenAI Python SDK ships a parse helper. Pass the Pydantic class directly as response_format, then read the parsed object from message.parsed.
from openai import OpenAI
client = OpenAI()
completion = client.chat.completions.parse(
model="gpt-5.6",
messages=[
{"role": "system", "content": "Extract the event information."},
{"role": "user",
"content": "Alice and Bob are going to a science fair on Friday."},
],
response_format=CalendarEvent,
)
event = completion.choices[0].message.parsed
print(event.name, event.participants) # real Python attributes
No json.loads, no try/except around a fragile string. event is already a CalendarEvent. If the provider cannot honor the schema it raises an error you can catch, instead of quietly returning a half-broken dict.
The same model works through the Responses API with client.responses.parse(...) and text_format=CalendarEvent. Functionally identical for this case.
At the raw REST level this maps to response_format: {type: "json_schema", json_schema: {name: "...", schema: {...}, strict: true}}. You almost never write that by hand; the SDK helper does it. Worth keeping in mind when someone pastes a curl example that looks enormous.
Step 3: the same schema on a local Ollama server
Running the model on your own machine changes nothing about the schema. Ollama's format parameter accepts the JSON schema directly.
from ollama import chat
response = chat(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Alice and Bob attend a science fair on Friday."}],
format=CalendarEvent.model_json_schema(),
options={"temperature": 0},
)
event = CalendarEvent.model_validate_json(response.message.content)
print(event)
Two differences worth noting. First, you pass format= with the generated schema, then validate the returned text yourself with model_validate_json. Second, set temperature to 0. Ollama's own docs recommend it: low temperature keeps the model on the schema rather than letting it drift. If you only need any valid object, format="json" is the lighter JSON-mode option, but you lose the field contract.
Step 4: a real extraction task
Structured output earns its keep when you extract a list of records from messy text. Same idea, one level of nesting.
class Pet(BaseModel):
name: str
animal: str
age: int
color: str | None = None
class PetList(BaseModel):
pets: list[Pet]
text = "Luna is a 5 year old grey cat who loves yarn. Loki, a 2 year old black cat, only chases tennis balls."
resp = chat(
model="llama3.1:8b",
messages=[{"role": "user", "content": text}],
format=PetList.model_json_schema(),
options={"temperature": 0},
)
data = PetList.model_validate_json(resp.message.content)
print(data.pets[0].name, data.pets[0].age)
Because pets is typed as list[Pet], a missing field raises ValidationError instead of silently producing a spare key. In an agent that is the right failure mode: you catch it, log it, and retry with a narrower prompt, rather than passing broken data downstream.
When to use structured outputs vs alternatives
- Structured outputs (schema) and function/tool calling both give typed results. Use function calling when the output is meant to trigger an action in your system. Use
response_formatwhen the model should produce data for the user, like a UI payload or an extraction row. - JSON mode (
format="json", orjson_object) still helps, but only guarantees parseable JSON. Treat it as a cheap upgrade over raw text, never as a replacement for a schema contract when types matter. - Prompt-only JSON works only for low stakes, single-shot asks where a malformed reply is harmless. It is the reason your regex exists. Drop it once the output feeds code.
Small models on constrained hardware handle shallow schemas better than deeply nested ones. Keep extraction contracts around two levels deep, and move to a bigger model if nested lists start coming back empty.
Next steps
- Read the official Structured Outputs guide for the full supported-schema list and the function-calling interplay.
- Check the Ollama structured outputs docs for the JavaScript (Zod) equivalent.
- Wire the same pattern into a LangChain agent with
with_structured_output()so the validated object flows straight into the next node.
The payoff is small in code and large in reliability. You delete the parser, you delete the regex, and the typed object just shows up. That is the whole job.