← Back to Blog
Oct 29, 2026 · 11 min read

A Typed AI Agent You Can Test Without an API Key

AI AgentsPydantic AIPythonTesting

An agent bug rarely arrives as a wrong answer. It arrives as a KeyError inside a tool, three retries that fail the same way, and a refund of 0.00 that nobody notices for a week.

The usual patch is one more sentence in the system prompt: "Always look up the order first and use the exact id." A prompt is a request. The build below turns the parts that matter into rules instead: a tool the model cannot call with junk, an output type checked before your code touches it, a refund cap enforced in Python, and a test suite that runs the entire agent loop with no API calls.

Pydantic AI 2.63.0 shipped on October 28, 2026. It puts a typed agent loop in front of any model provider, which means the type boundary sits exactly where these bugs live.

What you need

  • Python 3.10 or newer, since the examples use str | None
  • Ollama running locally, or an API key for a hosted provider
  • pytest for the last section

Step 1: install, and point the agent at a local model

uv venv
source .venv/bin/activate
uv pip install "pydantic-ai==2.63.0" pytest
ollama pull qwen3

Pydantic AI talks to Ollama through its OpenAI-compatible endpoint, so the full pydantic-ai package is enough. If you install pydantic-ai-slim instead, add the openai extra. The base URL defaults to http://localhost:11434/v1; override it with OLLAMA_BASE_URL when Ollama runs on another host.

Step 2: describe the data once

Two Pydantic models and one dataclass do most of the work here. Order is what the store returns, Decision is what the agent must produce, and SupportDeps is everything the tools are allowed to touch.

from dataclasses import dataclass, field
from datetime import date
from decimal import Decimal
from typing import Literal

from pydantic import BaseModel, Field
from pydantic_ai import Agent, ModelRetry, RunContext


class Order(BaseModel):
    id: str
    total: Decimal
    delivered_on: date | None = None
    refunded: bool = False


class Decision(BaseModel):
    action: Literal["refund", "replace", "deny"]
    amount: Decimal = Field(default=Decimal("0"), ge=0)
    reason: str = Field(max_length=200)


@dataclass
class SupportDeps:
    orders: dict[str, Order]
    today: date
    refund_cap: Decimal = Decimal("150.00")
    calls: list[str] = field(default_factory=list)


support_agent = Agent(
    "ollama:qwen3",
    deps_type=SupportDeps,
    output_type=Decision,
    instructions=(
        "You triage refund requests for an online store. "
        "Always look up the order before deciding. "
        "Never invent an order id. Keep reason under 200 characters."
    ),
)

The deps_type argument deserves a note: you pass the type, not an instance. Nothing is constructed at import time and nothing is stored globally. The instance arrives per run in run_sync(..., deps=deps), which is what makes the tests in step 6 short. You replace a dataclass, not a module.

Step 3: give the tool a way to say no

@support_agent.tool
def lookup_order(ctx: RunContext[SupportDeps], order_id: str) -> str:
    """Fetch one order by its id, for example ORD-1042."""
    ctx.deps.calls.append(order_id)
    order = ctx.deps.orders.get(order_id)
    if order is None:
        raise ModelRetry(
            f"No order with id {order_id!r}. Ask the customer for the id printed on the receipt."
        )
    return order.model_dump_json()

Use @agent.tool when the function needs ctx, and @agent.tool_plain when it does not. Raising ModelRetry sends your message back to the model as a retry prompt and lets it try again; it does not blow up the run. The message itself is the point, because it is the only feedback the model gets:

No order with id 'ORD-9999'. Ask the customer for the id printed on the receipt.

One specific instruction like that moves behavior further than any amount of "be careful" in the system prompt. Retries are budgeted, and @support_agent.tool(retries=2) raises the budget for a tool that legitimately needs a second attempt.

Step 4: validate the decision, not the vibe

output_type=Decision means the run cannot finish with prose. The model produces a tool call whose arguments have to satisfy Decision, and Pydantic rejects anything else before your code runs.

The validator is where business logic goes, because business logic is not a prompt:

@support_agent.output_validator
def enforce_refund_cap(ctx: RunContext[SupportDeps], decision: Decision) -> Decision:
    if decision.action == "refund" and decision.amount > ctx.deps.refund_cap:
        raise ModelRetry(
            f"Refunds above {ctx.deps.refund_cap} need a human. "
            "Use action='deny' and say the request was escalated."
        )
    if decision.action != "refund" and decision.amount != 0:
        raise ModelRetry("amount must be 0 unless action is refund.")
    return decision

Notice what it does not do: it does not ask the model to be honest about the cap. It compares two numbers. A 900.00 refund against a 150.00 cap raises ModelRetry, the model is told to deny and escalate, and the second attempt is validated the same way.

Decimal instead of float is deliberate. Money in binary floats turns into 24.900000000000002 in a reason field six months later.

Step 5: cap the loop before it caps your card

A retry loop that keeps failing is a bill. UsageLimits ends it, and capture_run_messages keeps the work that already happened.

from pydantic_ai import UsageLimits, capture_run_messages
from pydantic_ai.exceptions import UsageLimitExceeded


def handle(message: str, deps: SupportDeps) -> Decision | None:
    with capture_run_messages() as messages:
        try:
            result = support_agent.run_sync(
                message, deps=deps, usage_limits=UsageLimits(request_limit=4)
            )
            return result.output
        except UsageLimitExceeded as exc:
            print(f"stopped: {exc}")
    print(f"kept {len(messages)} messages for the next attempt")
    return None

The captured messages are a normal run history. Passing them back as message_history resumes from the last completed step instead of re-running tools with side effects, which in a support flow means no second refund.

Step 6: test the loop with zero network calls

This is the part that changes how the agent feels to maintain. No key, no latency, no assertions that break because the model rephrased a sentence.

from pydantic_ai import ModelResponse, ToolCallPart, capture_run_messages, models
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai.models.test import TestModel

models.ALLOW_MODEL_REQUESTS = False

That line is a tripwire. Any code path that tries to reach a real model raises instead of spending money, including the path you forgot to override.

TestModel covers the happy path. Two details matter. call_tools=[] stops it from calling every tool with generated junk, which a tool that rejects unknown ids would otherwise turn into a pile of retries. custom_output_args feeds exact arguments into the output tool, so the validator runs for real.

def test_happy_path_without_network(deps: SupportDeps) -> None:
    with support_agent.override(
        model=TestModel(
            call_tools=[],
            custom_output_args={"action": "deny", "amount": "0", "reason": "Out of policy."},
        )
    ):
        result = support_agent.run_sync("Can I get a refund on ORD-1042?", deps=deps)
    assert result.output.action == "deny"
    assert result.usage.requests == 1

FunctionModel covers the path that actually breaks in production, where the model is wrong first and correct later:

def scripted_model(messages, info: AgentInfo) -> ModelResponse:
    if len(messages) == 1:
        return ModelResponse(parts=[ToolCallPart("lookup_order", {"order_id": "ORD-9999"})])
    retry_seen = any(
        getattr(part, "part_kind", "") == "retry-prompt" for msg in messages for part in msg.parts
    )
    if retry_seen and not any(
        getattr(part, "tool_name", None) == "lookup_order"
        and getattr(part, "part_kind", "") == "tool-return"
        for msg in messages
        for part in msg.parts
    ):
        return ModelResponse(parts=[ToolCallPart("lookup_order", {"order_id": "ORD-1042"})])
    return ModelResponse(
        parts=[
            ToolCallPart(
                info.output_tools[0].name,
                {"action": "refund", "amount": "24.90", "reason": "Item arrived damaged."},
            )
        ]
    )


def test_tool_retry_then_success(deps: SupportDeps) -> None:
    with capture_run_messages() as messages:
        with support_agent.override(model=FunctionModel(scripted_model)):
            result = support_agent.run_sync("Refund please, order ORD-9999.", deps=deps)
    assert result.output.action == "refund"
    assert deps.calls == ["ORD-9999", "ORD-1042"]
    retries = [
        part
        for msg in messages
        for part in msg.parts
        if getattr(part, "part_kind", "") == "retry-prompt"
    ]
    assert len(retries) == 1

info.output_tools[0].name is how a scripted model finishes a structured run. With output_type set, the final answer arrives as a tool call, not as text. And capture_run_messages lets you assert that exactly one retry prompt was produced, which is the assertion that matters: the tool rejected the id once and the model corrected itself once.

Run it with pytest -q. Same result on your laptop and in CI, and the CI job needs no secrets.

Step 7: change the model without touching the logic

Model choice lives in one string. Swap ollama:qwen3 for openai:, anthropic:, google: or any other supported prefix, and the tools, validators, limits and tests stay exactly as they are.

For local models there is one upgrade worth knowing. Ollama 0.5.0 and newer enforces json_schema through llama.cpp's grammar-constrained decoder, and Pydantic AI exposes that as NativeOutput:

from pydantic_ai.output import NativeOutput

strict_agent = Agent(
    "ollama:qwen3",
    deps_type=SupportDeps,
    output_type=NativeOutput(Decision),
    instructions="You triage refund requests for an online store.",
)

The difference is where the constraint bites. Tool output asks the model to call a function with valid arguments and validates afterwards. NativeOutput constrains generation itself, so malformed JSON stops being a retry you pay for.

When this is the wrong tool

  • LangGraph when the control flow is the product: branches, checkpoints, approval steps as first-class nodes. If you are drawing the graph anyway, model it directly.
  • OpenAI Agents SDK when you are committed to OpenAI and handoffs between agents are the primitive you need.
  • Claude Agent SDK when the job is closer to a coding or file-editing agent with a sandbox already provided.
  • Pydantic AI when the agent is one component inside a typed Python service and you want its tests to look like the rest of the repo.

What unit tests do not tell you

They prove the code around the model. They say nothing about whether the answers are good. That is evals, with their own harness and their own budget, and mixing the two is how teams spend an afternoon debugging a prompt when the bug was a missing strip().

The next step that pays off immediately: put pytest in CI with ALLOW_MODEL_REQUESTS = False, so a forgotten override fails the build instead of hitting a paid endpoint at 2am. Then write five eval cases for the decisions that matter, starting with the refund above the cap, because that is the one your validator is not allowed to get wrong.

References

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation