You want a local AI assistant that can actually do things: read files, run shell commands, search the web. Not a chatbot that talks about doing things, but one that executes them.
Most AI agent tutorials start with a cloud API. You send your code, your data, and your prompts to someone else's server. For many use cases, that is fine. For personal tooling, codebases with secrets, or air-gapped environments, it is not.
Ollama added tool calling support in v0.3.0. Combined with the Python SDK, you can build an agent that runs entirely on your machine, decides which tools to call, executes them, and feeds the results back to the model until the task is done. No API key needed. No data leaves your laptop.
This tutorial walks through building one from scratch. By the end, you will have a working CLI agent that can read and write files, run shell commands, and handle multi-step requests.
Prerequisites
- Python 3.10 or newer
- Ollama installed (ollama.com)
- 8GB RAM minimum (16GB recommended for Qwen 3 8B)
- Terminal access
Step 1: Install Ollama and Pull a Model
# Install Ollama (Linux/macOS)
curl -fsSL https://ollama.com/install.sh | sh
# Pull a model that supports tool calling
ollama pull qwen3:8b
Qwen 3 is the most reliable model for tool calling through Ollama right now. It has native tool calling in its chat template, handles parallel tool calls, and its think=True mode shows you the model's reasoning. If you have limited RAM, qwen3:4b works but makes more mistakes in tool selection.
For comparison, here is how popular models stack up for tool calling:
| Model | RAM | Tool calling | Notes |
|---|---|---|---|
qwen3:8b |
8GB | Native | Best balance for most setups |
qwen3:4b |
4GB | Native | Lighter, less reliable |
llama3.1:8b |
8GB | Native | Good, wide framework support |
gemma4:9b |
12GB | Native | Strong if you have the RAM |
Verify the model works:
ollama run qwen3:8b "Say hello in one sentence"
Step 2: Set Up the Project
mkdir ollama-agent && cd ollama-agent
python3 -m venv venv
source venv/bin/activate
# Install the Ollama Python SDK
pip install ollama
Create the main file:
touch agent.py
Step 3: Define Your Tools
Tools are regular Python functions with type hints and docstrings. The Ollama SDK reads these to build JSON schemas that the model uses to decide which tool to call.
Open agent.py and add the tool definitions:
import os
import subprocess
from pathlib import Path
WORKSPACE = Path.home() / "agent-workspace"
WORKSPACE.mkdir(exist_ok=True)
def safe_path(filepath: str) -> str:
"""Resolve a path and ensure it stays inside the workspace."""
resolved = (WORKSPACE / filepath).resolve()
if not str(resolved).startswith(str(WORKSPACE.resolve())):
return "ERROR: path escapes workspace"
return str(resolved)
def read_file(filepath: str) -> str:
"""Read the contents of a file.
Args:
filepath: Relative path inside the workspace
Returns:
The file contents, or an error message
"""
path = safe_path(filepath)
if path.startswith("ERROR"):
return path
try:
return Path(path).read_text()
except FileNotFoundError:
return f"ERROR: file not found: {filepath}"
except Exception as e:
return f"ERROR: {e}"
def write_file(filepath: str, content: str) -> str:
"""Write content to a file, creating parent directories if needed.
Args:
filepath: Relative path inside the workspace
content: The content to write
Returns:
Confirmation message
"""
path = safe_path(filepath)
if path.startswith("ERROR"):
return path
try:
Path(path).parent.mkdir(parents=True, exist_ok=True)
Path(path).write_text(content)
return f"OK: wrote {len(content)} bytes to {filepath}"
except Exception as e:
return f"ERROR: {e}"
def run_command(command: str) -> str:
"""Run a shell command and return its output.
Args:
command: The shell command to execute
Returns:
Combined stdout and stderr
"""
# Block dangerous commands
blocked = ["rm -rf", "mkfs", "dd if=", "> /dev/", ":(){ :|:& };:"]
for pattern in blocked:
if pattern in command:
return f"ERROR: blocked dangerous command pattern: {pattern}"
try:
result = subprocess.run(
command,
shell=True,
capture_output=True,
text=True,
timeout=30,
cwd=str(WORKSPACE),
)
output = result.stdout
if result.stderr:
output += f"
STDERR: {result.stderr}"
return output if output else "(no output)"
except subprocess.TimeoutExpired:
return "ERROR: command timed out after 30 seconds"
except Exception as e:
return f"ERROR: {e}"
def list_files(directory: str = ".") -> str:
"""List files and directories at a given path.
Args:
directory: Relative directory path inside the workspace (default: current dir)
Returns:
Newline-separated list of files and directories
"""
path = safe_path(directory)
if path.startswith("ERROR"):
return path
try:
entries = sorted(Path(path).iterdir())
lines = []
for entry in entries:
prefix = "d " if entry.is_dir() else "f "
lines.append(f"{prefix}{entry.name}")
return "
".join(lines) if lines else "(empty directory)"
except FileNotFoundError:
return f"ERROR: directory not found: {directory}"
except Exception as e:
return f"ERROR: {e}"
The safe_path function is important. It keeps all file operations inside the workspace directory. Without it, the model could read or write files anywhere on your system. The run_command function blocks obviously dangerous shell patterns and has a 30-second timeout.
Step 4: Build the Agent Loop
The agent loop is the core pattern. Send a message to the model. If it returns tool calls, execute them, add the results to the conversation, and send it back. Repeat until the model gives a text response.
from ollama import chat, ChatResponse
MODEL = "qwen3:8b"
MAX_ITERATIONS = 10
TOOLS = {
"read_file": read_file,
"write_file": write_file,
"run_command": run_command,
"list_files": list_files,
}
def agent_run(user_input: str, messages: list[dict] | None = None) -> str:
"""Run the agent loop for a single user request.
Args:
user_input: The user's request
messages: Optional conversation history (for multi-turn)
Returns:
The agent's final text response
"""
if messages is None:
messages = []
messages.append({"role": "user", "content": user_input})
for iteration in range(MAX_ITERATIONS):
response: ChatResponse = chat(
model=MODEL,
messages=messages,
tools=list(TOOLS.values()),
options={"temperature": 0.1},
think=True,
)
messages.append(response.message)
# Show thinking if available
if response.message.thinking:
print(f"\033[90m[thinking] {response.message.thinking[:200]}...\033[0m")
# No tool calls means the model is done
if not response.message.tool_calls:
return response.message.content
# Execute each tool call
for call in response.message.tool_calls:
fn_name = call.function.name
fn_args = call.function.arguments or {}
if fn_name not in TOOLS:
result = f"ERROR: unknown tool: {fn_name}"
else:
try:
result = TOOLS[fn_name](**fn_args)
except Exception as e:
result = f"ERROR: {e}"
print(f"\033[33m -> {fn_name}({fn_args}) -> {result[:100]}\033[0m")
messages.append({
"role": "tool",
"tool_name": fn_name,
"content": str(result),
})
return "ERROR: agent exceeded maximum iterations"
A few things to note:
temperature: 0.1keeps tool selection deterministic. Higher values cause the model to sometimes pick the wrong tool or hallucinate arguments.think=Trueshows the model's reasoning chain. Useful for debugging when it calls the wrong tool.- The 10-iteration limit prevents infinite loops. If the model keeps calling tools without reaching a conclusion, the loop stops.
- Tool results are added as
role: "tool"messages so the model sees what happened and can decide what to do next.
Step 5: Add the CLI
Now wire it up with a REPL that maintains conversation history:
import sys
def main():
print("Ollama CLI Agent")
print(f"Model: {MODEL}")
print(f"Workspace: {WORKSPACE}")
print("Commands: /clear, /history, /quit
")
messages: list[dict] = []
# System prompt to set behavior
messages.append({
"role": "system",
"content": (
"You are a helpful CLI assistant. You have access to file and "
"shell tools. Always use the tools to accomplish tasks rather "
"than just describing what to do. When reading or writing files, "
"use relative paths within the workspace. Keep responses concise."
),
})
while True:
try:
user_input = input("\033[36m>\033[0m ").strip()
except (EOFError, KeyboardInterrupt):
print("
Bye.")
break
if not user_input:
continue
# Handle slash commands
if user_input == "/clear":
messages = messages[:1] # keep system prompt
print("Conversation cleared.")
continue
if user_input == "/history":
for msg in messages[1:]: # skip system prompt
role = msg.get("role", "?")
content = msg.get("content", "")[:100]
print(f" [{role}] {content}")
continue
if user_input in ("/quit", "/exit"):
print("Bye.")
break
response = agent_run(user_input, messages)
print(f"
\033[32m{response}\033[0m
")
if __name__ == "__main__":
main()
Step 6: Test It
Run the agent:
python agent.py
Try these commands to see it in action:
> Create a file called notes.txt with today's ideas for a side project
The model should call write_file with a filename and content, then confirm.
> What files are in the workspace?
It should call list_files and show you notes.txt.
> Read notes.txt and add a new idea at the end
This is a multi-step request. The model should call read_file first, then call write_file with the updated content. Watch the tool calls execute in sequence.
> Run "uname -a" and tell me what OS I'm on
It should call run_command and summarize the output.
How the Agent Loop Works
Here is what happens behind the scenes for a request like "Read notes.txt and add a new idea":
- You send the message to the model
- The model decides it needs to read the file first. It returns a tool call:
read_file("notes.txt") - Your code executes the tool and sends the result back
- The model sees the file contents. It decides to write the updated version. It returns:
write_file("notes.txt", "...updated content...") - Your code executes the write
- The model returns a text response: "Added your new idea to notes.txt"
Each step is a separate API call. The model always sees the full conversation, including all previous tool calls and their results, so it can make informed decisions about what to do next.
Extending the Agent
The four tools here are a starting point. Here are practical additions:
Web search (requires a search API):
import urllib.request
import json
def web_search(query: str) -> str:
"""Search the web using DuckDuckGo.
Args:
query: The search query
Returns:
Search results as text
"""
url = f"https://api.duckduckgo.com/?q={query}&format=json&no_html=1"
try:
with urllib.request.urlopen(url, timeout=10) as resp:
data = json.loads(resp.read())
results = []
if data.get("Abstract"):
results.append(data["Abstract"])
for topic in data.get("RelatedTopics", [])[:5]:
if isinstance(topic, dict) and "Text" in topic:
results.append(topic["Text"])
return "
".join(results) if results else "No results found"
except Exception as e:
return f"ERROR: {e}"
Git operations (safe, read-only by default):
def git_status() -> str:
"""Show git status of the workspace.
Returns:
The git status output
"""
try:
result = subprocess.run(
["git", "status", "--short"],
capture_output=True, text=True, cwd=str(WORKSPACE), timeout=10
)
return result.stdout or "(clean working tree)"
except Exception as e:
return f"ERROR: {e}"
Add these to the TOOLS dict and they become available to the model automatically:
TOOLS = {
"read_file": read_file,
"write_file": write_file,
"run_command": run_command,
"list_files": list_files,
"web_search": web_search,
"git_status": git_status,
}
Model Selection Tips
Tool calling quality varies significantly between models. Some patterns to watch for:
- Qwen 3 handles parallel tool calls well. When it needs data from multiple sources, it sends all the tool calls in one response instead of doing them one by one.
- Llama 3.1 works but occasionally passes incorrect argument types (string instead of integer). Double-check tool inputs if you see type errors.
- Smaller models (3B parameters and below) frequently call the wrong tool or hallucinate tool names that do not exist. Stick with 8B or larger for tool calling.
If the model struggles with your tools, make the docstrings more explicit. Describe not just what each argument is, but what values are valid. Models pay close attention to docstrings when deciding how to call tools.
Common Issues
"Model does not support tools": Your model version might be outdated. Pull the latest: ollama pull qwen3:8b. Tool calling support requires Ollama v0.3.0 or newer.
Model calls tools in a loop: This happens when the model keeps requesting information it already has. Add a check in the agent loop: if the same tool with the same arguments is called twice in a row, force a text response.
Slow responses: Tool calling adds latency because each iteration is a separate model inference. On CPU-only setups with 8B models, expect 5-15 seconds per tool call. GPU acceleration cuts this to 1-3 seconds.
Model ignores tools: Some system prompts cause the model to describe what it would do instead of actually calling tools. The system prompt in this tutorial explicitly tells the model to use tools. Adjust if you change it.
Where to Go Next
- MCP integration: Wrap these tools as an MCP server so other AI clients (Claude Desktop, Cursor) can use them too. See the MCP server tutorial in this blog.
- Persistent memory: Store facts the model learns about your workspace in a local SQLite database. Re-inject them as system context on startup.
- Multi-agent setup: Use one agent for planning and another for execution. Ollama supports running multiple models simultaneously.
- Web UI: Connect the agent to Open WebUI for a browser-based interface with the same tool calling backend.