← Back to Blog
Jul 03 2026 · 8 min read

Build a Local AI Agent with Ollama and Function Calling

PythonOllamaAI AgentsLocal LLMTutorialTools

Cloud LLM APIs are convenient, but they cost money, log your prompts, and need a network connection. For some tasks — internal tools, offline development, anything that touches sensitive data — running the model locally is a better fit. The catch: getting an agent to actually call functions correctly on a local model is fiddlier than the docs make it sound.

This walks through a working setup: Ollama running a model that supports tool use, a small Python script, and one real tool (reading a local file). The whole thing fits in under 100 lines.

Prerequisites

  • macOS, Linux, or Windows with WSL2
  • 16GB RAM minimum (32GB recommended for 13B+ models)
  • Python 3.10+
  • About 10GB of disk for model weights

Step 1: Install Ollama

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Verify
ollama --version

That's it. Ollama runs as a local service on http://127.0.0.1:11434 and exposes an OpenAI-compatible API. No account, no login.

Step 2: Pick a Model That Supports Tools

Not every model handles function calling well. The ones that do, as of early 2026:

Model Size Tool Support Quality
llama3.1:8b 4.7GB Good Strong general purpose
qwen2.5:14b 9.0GB Very good Better at structured output
mistral-nemo:12b 7.1GB Good Strong reasoning
llama3.3:70b 43GB Excellent Needs 64GB+ RAM

Pull one:

ollama pull llama3.1:8b

The 8B model is the sweet spot for most laptops. The 70B models are noticeably smarter but you need serious hardware.

Step 3: Verify the Model Responds

ollama run llama3.1:8b "What is 17 * 23?"

If you get 391, the model is working.

Step 4: The Python Loop

A tool-calling agent is a loop. The model sees a prompt and a list of available tools. If it decides to call a tool, the loop runs the tool, feeds the result back, and asks the model again. This repeats until the model produces a final answer (or hits a turn limit).

import json
import requests

OLLAMA_URL = "http://127.0.0.1:11434/api/chat"
MODEL = "llama3.1:8b"

# Tool the agent can call. In a real project, put each tool in its own function
# and dispatch by name. One tool is enough to show the pattern.
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "description": "Read the contents of a text file at the given path. Use this when the user asks about a file.",
            "parameters": {
                "type": "object",
                "properties": {
                    "path": {
                        "type": "string",
                        "description": "Absolute or relative path to the file",
                    }
                },
                "required": ["path"],
            },
        },
    }
]


def read_file(path: str) -> str:
    try:
        with open(path, "r", encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        return f"Error: file not found at {path}"
    except Exception as e:
        return f"Error: {e}"


TOOL_FUNCTIONS = {"read_file": read_file}


def chat(messages: list, max_turns: int = 5) -> str:
    for turn in range(max_turns):
        response = requests.post(
            OLLAMA_URL,
            json={
                "model": MODEL,
                "messages": messages,
                "tools": TOOLS,
                "stream": False,
            },
        )
        response.raise_for_status()
        data = response.json()
        msg = data["message"]

        # If the model did not call a tool, it gave a final answer.
        if not msg.get("tool_calls"):
            return msg["content"]

        # Otherwise, run each tool call and append the result.
        messages.append(msg)
        for call in msg["tool_calls"]:
            name = call["function"]["name"]
            args = call["function"]["arguments"]
            result = TOOL_FUNCTIONS[name](**args)
            messages.append(
                {
                    "role": "tool",
                    "content": str(result),
                }
            )

    return "Agent stopped after max turns without a final answer."


if __name__ == "__main__":
    messages = [
        {
            "role": "system",
            "content": "You are a helpful assistant. Use the read_file tool when the user asks about a file.",
        },
        {
            "role": "user",
            "content": "Read ./README.md and tell me what this project does in one sentence.",
        },
    ]
    print(chat(messages))

Run it with any README in the working directory:

python agent.py

The model receives the user request, decides to call read_file, gets the file contents, and answers in plain English. No API key, no cost per token.

Step 5: Add a Second Tool

The loop already handles multiple tool calls per turn. Add another tool the same way:

def list_directory(path: str) -> str:
    import os
    try:
        entries = os.listdir(path)
        return json.dumps(entries)
    except Exception as e:
        return f"Error: {e}"


TOOL_FUNCTIONS["list_directory"] = list_directory
TOOLS.append({
    "type": "function",
    "function": {
        "name": "list_directory",
        "description": "List files and folders at the given path. Use this when the user wants to know what is in a directory.",
        "parameters": {
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "Directory path"}
            },
            "required": ["path"],
        },
    },
})

The model can now combine tools: list_directory first, then read_file on something it found. This is the basic shape of every agent framework — LangChain, LlamaIndex, AutoGen. They all wrap this loop with better prompts, retries, and tracing.

What Goes Wrong and How to Fix It

The model hallucinates a tool call. Smaller models sometimes invent tools that are not in the list. Tighten the system prompt: "Only use the tools listed. If none fit, say so."

Tool arguments are wrong types. A common one: the model passes {"path": null} or forgets a required field. Validate inside the tool function and return a clear error string. The model reads errors and retries.

Latency. On a 2020 MacBook Air, llama3.1:8b takes 1-3 seconds per token. A 5-turn conversation can feel slow. Drop to llama3.2:3b (2GB) for prototyping — much faster, weaker reasoning.

Model ignores the tool. Some prompts work better than others. "Read the README and summarize" works. "Tell me about the project" does not — the model has no idea what you mean. Be specific about when to use each tool.

When This Is Enough

For a single user, single machine, a couple of tools: this loop is the entire agent. No framework needed. Adding error handling, retries, and logging is straightforward. You can ship this.

When you need multiple agents, shared state, async, observability, or production reliability, switch to LangChain, LangGraph, or the Claude Agent SDK. They are not magic — they wrap the same loop. Knowing how the loop works makes the framework code make sense.

Where to Go Next

  • Add structured output with format parameter in Ollama (JSON schema constrained generation)
  • Try qwen2.5:14b for better tool calling accuracy
  • Wrap tools in an MCP server so the same tools work from Claude Desktop, Cursor, or any MCP host
  • Add a vector store (Chroma or Qdrant) for RAG — read files, chunk them, embed, retrieve

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation