Run LLMs Locally with Ollama: From Install to a Production-Ready API

You don't need to send every prompt to a third-party API. Some workloads, like sensitive data, high-volume batch jobs, or anything that needs to keep running offline, are better served by a model that runs on your own machine. Ollama is the easiest way to get there. It packages llama.cpp behind a clean REST API, ships binary installers for macOS, Linux, and Windows, and exposes an OpenAI-compatible endpoint at http://localhost:11434/v1/chat/completions so existing tools work without modification.

This is a hands-on walkthrough. By the end, you'll have Ollama running, a quantized model pulled, a custom Modelfile built, and a small agent-style script that combines chat, structured output, and embeddings from a single HTTP server.

Prerequisites

• A machine with at least 8 GB of RAM (16 GB is more comfortable for 7B-class models)
• macOS, Linux, or Windows (WSL2 works on Windows)
• About 5 GB of free disk for a single 7B model
• curl and jq for testing the API

Check the official install page for the most current command for your OS. On Linux the one-liner is:

curl -fsSL https://ollama.com/install.sh | sh

On macOS you can use the download from the website or brew install ollama. The Windows installer is a regular .exe from the download page.

After install, verify the server is up:

ollama --version
systemctl status ollama    # Linux only; on macOS the app launches the server
curl http://localhost:11434/api/version

The last command returns something like {"version":"0.6.0"}. If you get a connection refused, the daemon isn't running yet. On Linux with systemd, sudo systemctl start ollama. On macOS, launch the Ollama app once so it registers as a background service.

Step 1: Pull a model

ollama pull downloads a quantized GGUF from the Ollama registry. The library at ollama.com/library lists what's available. For a first run, llama3.2:3b is a sensible default — small enough to run on a laptop, big enough to be useful.

ollama pull llama3.2:3b

This downloads roughly 2 GB. While you're at it, pull an embedding model — they are small and you'll want one for retrieval work later.

ollama pull nomic-embed-text
ollama pull all-minilm

List what's local:

ollama list

NAME                    ID          SIZE      MODIFIED
llama3.2:3b             a80c4f17acd5 2.0 GB    2 minutes ago
nomic-embed-text:latest 0a109f422b47 274 MB    2 minutes ago
all-minilm:latest       1b226e2802db 46 MB     2 minutes ago

The model stays loaded in memory for 5 minutes by default after the last request. You can tune this with the keep_alive parameter, which I'll cover in the API section.

Quick sanity check from the CLI:

ollama run llama3.2:3b "Explain the difference between TCP and UDP in two sentences."

The ollama run command starts an interactive REPL if you don't pass a prompt. Type /bye to exit.

Step 2: Call the REST API

Everything ollama run does is just a thin wrapper around HTTP. The two endpoints you need are POST /api/generate (single prompt) and POST /api/chat (multi-turn). The full spec lives at github.com/ollama/ollama/blob/main/docs/api.md.

/api/generate for a one-shot completion:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

You get back a single JSON object when stream is false. With stream: true (the default), the response is a stream of newline-delimited JSON chunks, one per token. The last chunk has "done": true and includes timing fields:

• total_duration: wall clock for the request
• load_duration: time spent loading the model into memory
• prompt_eval_count: tokens in your prompt
• eval_count: tokens generated
• eval_duration: time spent generating

To compute tokens per second, the formula from the docs is eval_count / eval_duration * 10^9. Useful when you're comparing quantizations.

/api/chat is what you want for conversation. The messages array takes system, user, and assistant roles:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "stream": false,
  "messages": [
    {"role": "system", "content": "You answer in exactly one sentence."},
    {"role": "user", "content": "What is the capital of France?"}
  ]
}'

Pass tools to enable tool calling on models that support it. The model returns a tool_calls array; you execute the function, then send the result back as a tool role message. The tool-capable model list is searchable at ollama.com.

Step 3: Use the OpenAI-compatible endpoint

This is the part that makes Ollama drop into existing stacks. Set base_url to http://localhost:11434/v1 and most OpenAI clients just work. No code change beyond the URL.

Python with the official openai library:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by the client, ignored by Ollama
)

resp = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Write a haiku about caching."}],
)
print(resp.choices[0].message.content)

Node.js:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama",
});

const resp = await client.chat.completions.create({
  model: "llama3.2:3b",
  messages: [{ role: "user", content: "Write a haiku about caching." }],
});
console.log(resp.choices[0].message.content);

The same pattern works with the JS, Go, and Rust SDKs. Anything that points at api.openai.com will work when you point it at the local server. This means LangChain, LlamaIndex, Hermes Agent, and any tool that speaks the OpenAI Chat Completions API can run against local models with one config change.

Step 4: Get structured output

Free-form text is fine for chat, but if you're building an agent you need predictable shapes. Pass a JSON schema in the format field and the model will generate output that conforms to it. The schema format is standard JSON Schema 2020-12.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Extract the name, age, and city from: Anna is 31 and lives in Berlin.",
  "stream": false,
  "format": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "age": {"type": "integer"},
      "city": {"type": "string"}
    },
    "required": ["name", "age", "city"]
  }
}'

The response field comes back as a JSON string that parses cleanly. In Python you can do the round trip in a few lines:

import json, urllib.request

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "city": {"type": "string"},
    },
    "required": ["name", "age", "city"],
}

req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=json.dumps({
        "model": "llama3.2:3b",
        "prompt": "Extract: Anna is 31 and lives in Berlin.",
        "stream": False,
        "format": schema,
    }).encode(),
    headers={"Content-Type": "application/json"},
)
data = json.loads(urllib.request.urlopen(req).read())
print(json.loads(data["response"]))
# {'name': 'Anna', 'age': 31, 'city': 'Berlin'}

Structured output is the piece that makes local models genuinely useful for pipelines. Skip the regex post-processing.

Step 5: Generate embeddings

Embeddings come from a separate endpoint and a separate model. Don't try to get them from a chat model — the output is not the same and the dimensions are not what you want.

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "Ollama runs large language models locally"
}'

The response is a list of float vectors, one per input string. To embed multiple texts at once, pass an array:

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": ["first text", "second text", "third text"]
}'

nomic-embed-text produces 768-dimensional vectors; all-minilm is 384. The dimensions parameter is supported and lets you truncate, but only certain models honor it. Use a smaller embed model if you care about vector DB cost and don't need the extra dimensions.

The truncate parameter (default true) automatically clips inputs that exceed the context window. Set it to false if you want to detect overflow yourself rather than get a silent truncation.

Step 6: Build a custom Modelfile

A Modelfile is the recipe for a customized model. You start from a base, set parameters, and inject a system prompt. This is how you get a consistent persona or a domain-specific tone without rewriting the system message on every call.

# Modelfile
FROM llama3.2:3b

# Lower temperature for more deterministic output
PARAMETER temperature 0.2

# Expand the context window if you have the RAM
PARAMETER num_ctx 8192

# Stop sequences that match the base model's chat template
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"

# Persistent system prompt baked into the model
SYSTEM You are a senior backend engineer who reviews pull requests. You respond in three short bullets. You never start with "Certainly" or "Great question". You flag missing tests, race conditions, and unclear naming. You do not flatter the author.

Save it as Modelfile (no extension) and build it:

ollama create pr-reviewer -f ./Modelfile
ollama run pr-reviewer "Review this PR title: 'Refactor user service'"

To see what was baked in:

ollama show pr-reviewer --modelfile

The Modelfile spec also supports TEMPLATE for custom chat templates, ADAPTER for LoRA adapters, LICENSE for legal text, and MESSAGE to seed conversation history. For 90% of customization work, FROM, PARAMETER, and SYSTEM are enough.

Step 7: Wire it into a small agent

Putting it all together — a script that takes a user question, retrieves a few relevant docs by embedding similarity, asks the chat model to answer with citations, and returns the result as a structured JSON object. The embedding model and the chat model can be different.

import json
import urllib.request
import numpy as np

OLLAMA = "http://localhost:11434"

DOCS = [
    "Ollama runs llama.cpp under the hood and exposes a REST API.",
    "The OpenAI-compatible endpoint lives at /v1/chat/completions.",
    "Structured output is enabled by passing a JSON schema in the format field.",
    "Modelfiles let you set parameters and a system prompt persistently.",
]


def post(path, body):
    req = urllib.request.Request(
        f"{OLLAMA}{path}",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(req).read())


def embed(texts):
    r = post("/api/embed", {"model": "nomic-embed-text", "input": texts})
    return np.array(r["embeddings"])


def retrieve(query, k=2):
    q = embed([query])[0]
    docs = embed(DOCS)
    scores = docs @ q / (np.linalg.norm(docs, axis=1) * np.linalg.norm(q))
    top = np.argsort(scores)[::-1][:k]
    return [DOCS[i] for i in top]


def answer(query):
    context = "\n".join(f"- {d}" for d in retrieve(query))
    prompt = f"Use the context to answer the question.\n\nContext:\n{context}\n\nQuestion: {query}"
    schema = {
        "type": "object",
        "properties": {
            "answer": {"type": "string"},
            "sources": {"type": "array", "items": {"type": "string"}},
        },
        "required": ["answer", "sources"],
    }
    r = post("/api/generate", {
        "model": "llama3.2:3b",
        "prompt": prompt,
        "stream": False,
        "format": schema,
    })
    return json.loads(r["response"])


if __name__ == "__main__":
    print(json.dumps(answer("How do I get structured output from Ollama?"), indent=2))

Run it with python3 agent.py. The model picks the two most relevant docs, writes an answer grounded in them, and returns a JSON object with the answer and the source list. No framework needed — the API is small enough that the glue is shorter than the imports.

Performance tuning

Three knobs move the needle on throughput and memory.

Quantization. The tags in the Ollama library encode the quantization level. q4_K_M is the recommended default — close to full quality at a fraction of the size. q8_0 is higher quality, larger. Full precision (f16) is rarely worth it for local inference. You can see what you're running with ollama show llama3.2:3b and looking at the size column.

Context window (num_ctx). The default is 2048 for most models. Larger contexts eat more RAM and slow down generation, sometimes dramatically. Set it as low as you can get away with. If your prompt plus expected output fits in 1024 tokens, set num_ctx 1024.

Keep alive. Default is 5 minutes. If you have bursty traffic, bump it up so the model doesn't reload between requests:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "hi",
  "stream": false,
  "keep_alive": "30m"
}'

0 unloads immediately. Negative values are documented as "keep forever" but you'll run out of RAM.

When to use Ollama vs hosted APIs

Ollama is the right call when:

• The data can't leave the machine for compliance reasons
• Latency to a hosted endpoint is too high (on-device voice, edge sensors)
• You're doing batch processing and the per-token cost of hosted APIs adds up
• You want to prototype without worrying about rate limits

A hosted API is the right call when:

• The model you need is much larger than your hardware can run (70B+ on a laptop)
• You need the absolute best quality and are willing to pay for the top model
• You're already paying for the API and the marginal cost of another request is near zero

The honest middle ground: use Ollama for development and CI, switch to a hosted endpoint for the heaviest production queries. Because the API surface is identical, this is a one-line change in most clients.

Where to go next

• The full API reference at github.com/ollama/ollama/blob/main/docs/api.md — every parameter, every endpoint, every example.
• The Modelfile spec at github.com/ollama/ollama/blob/main/docs/modelfile.mdx for deeper customization.
• The OpenAI compatibility post at ollama.com/blog/openai-compatibility for the list of supported fields and known gaps.
• The model library at ollama.com/library to see what's available, including vision, code, and embedding models.

References

• Ollama API documentation
• Modelfile reference
• OpenAI compatibility announcement