← Back to Blog
Nov 5, 2026 · 12 min read

Fix Ollama's Context Length Before Your Local Agent Starts Forgetting

OllamaLocal LLMAI AgentsPerformanceTutorial

A local agent runs fine for ten turns, then starts repeating itself, drops earlier instructions, or answers as if the first half of the conversation never happened. The model gets blamed first. Most of the time the problem is the context window: Ollama allocated a few thousand tokens, the conversation outgrew it, and the history got cut before it reached the model.

The confusion comes from two different numbers that both look like "context length". ollama show prints the maximum the model was trained for, which is often 32k or more. ollama ps prints what the server actually allocated on this machine, which may be 4k. The gap between those two numbers is where local agents lose their memory.

This guide closes that gap: read the real allocation, pin a budget per model, shrink the KV cache, and confirm each change with numbers from your own hardware.

What you need

  • Ollama installed and running (ollama --version)
  • One tool-capable model, for example qwen3:8b
  • curl and Python 3
  • Enough VRAM or RAM for the context you plan to use

Step 1: Read the context you actually got

Pull a model, send one request so it loads, then look at what the server did:

ollama pull qwen3:8b
ollama run qwen3:8b "say hi" > /dev/null
ollama ps

The output has a CONTEXT column and a PROCESSOR column. This is the example from the Ollama docs, and the numbers on your machine will differ:

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:latest    c6eb396dbd59    9.6 GB    100% GPU     131072     2 minutes from now

CONTEXT is the allocation. PROCESSOR tells you whether the model and its cache fit on the GPU. Now compare it with the model card:

ollama show qwen3:8b

That prints the context length the model was trained with. It is a ceiling, not a promise. The server picks its own number based on available VRAM. According to the Ollama docs, the default is tiered: under 24 GiB of VRAM gets 4k, 24 to 48 GiB gets 32k, and 48 GiB or more gets 256k. The FAQ still lists 4096 as the default, so the two pages disagree. Trust ollama ps, because that reflects the running process.

The same data is available over HTTP:

curl -s http://localhost:11434/api/ps | python3 -m json.tool

Each loaded model reports context_length, size, and size_vram. If size_vram is smaller than size, part of the model sits in system RAM and generation will crawl.

Step 2: Raise the server default

On Linux with the systemd service, edit the drop-in rather than the unit file:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Send a request, then check /api/ps again. context_length should read 32768. If it still says 4096, the environment variable never reached the process:

systemctl show ollama -p Environment

On macOS, the app reads launchd variables. Set it, then quit and reopen Ollama:

launchctl setenv OLLAMA_CONTEXT_LENGTH 32768

This number is global. It applies to the 1B autocomplete model too, and the KV cache for that model at 32k is VRAM you did not need to spend. That is what the next step is for.

Step 3: Pin a budget per model

A Modelfile lets each model carry its own defaults, so a shared server setting cannot quietly downgrade your coding agent.

FROM qwen3:8b
PARAMETER num_ctx 32768
PARAMETER temperature 0.7
PARAMETER repeat_penalty 1.05
SYSTEM """You are a coding assistant. Answer with patches, not essays."""

ollama create qwen3-agent -f Modelfile
ollama run qwen3-agent "list three uses of a ring buffer"
ollama ps

ollama ps should now show 32768 for the derived model. You can always inspect what got baked in:

ollama show --modelfile qwen3-agent | head -20

Keep separate models for separate jobs. A chat model at 8k, a coding model at 32k, an embedding model with no context budget worth thinking about. One global number forces every workload into the same envelope.

One caveat: a client that sends options.num_ctx explicitly in /api/chat or /api/generate overrides the Modelfile. If your agent framework sets its own value, the Modelfile is decoration. Check the request it sends before blaming the server.

Step 4: Shrink the cache instead of the context

KV cache size grows with context length. Flash attention plus a quantized cache is the cheapest way to keep a long context without buying a bigger GPU. Both are server-level environment variables:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -u ollama -n 50 --no-pager | grep -iE "flash attention|kv_cache_type"

The log should mention flash attention and the cache type. If it does not, the setting did not apply.

Per the docs, q8_0 uses roughly half the memory of f16 with a very small precision loss, and q4_0 uses roughly a quarter with a loss that becomes noticeable at longer contexts. Quantized cache requires flash attention; without it Ollama falls back to f16. And the setting is global for every model the server loads, so you cannot mix cache types in one process.

Quality impact is model-dependent. The docs call out models with high GQA counts, Qwen2 among them, as more sensitive to cache quantization. If you want q4_0 for the extra headroom, run your eval prompts against both settings before shipping it.

Step 5: Do the parallelism math

Parallel requests multiply the context allocation. The docs give the example directly: a 2K context with 4 parallel requests becomes an 8K context, with the memory that implies. Required memory scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH.

So a 32k context with four parallel slots is 128k of cache to allocate. On a 12 GB card that usually ends as a CPU split, and a CPU split is slower than a queue.

For a single-user local agent:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

If several people share one box, raise OLLAMA_NUM_PARALLEL to 2 or 4 after checking memory headroom. Requests beyond capacity queue up to OLLAMA_MAX_QUEUE (512 by default) and then the API returns 503, which is at least an honest error instead of a timeout.

Step 6: Stop paying the load cost

Models unload after 5 minutes of inactivity by default. An agent that runs every 15 minutes pays the load price on every run, and you can see that price in load_duration on each response.

[Service]
Environment="OLLAMA_KEEP_ALIVE=30m"

You can also set it per request, which wins over the server default:

curl http://localhost:11434/api/generate -d '{"model": "qwen3-agent", "keep_alive": "30m"}'
curl http://localhost:11434/api/generate -d '{"model": "qwen3-agent", "keep_alive": -1}'

An empty prompt preloads the model. A negative value keeps it resident until you free it yourself:

ollama stop qwen3-agent

Long keep-alive is great on a workstation and terrible on a shared GPU host where other jobs need the memory. Pick per machine, not per tutorial.

Step 7: Measure with the API, not with vibes

Every response from /api/generate includes load_duration, prompt_eval_count, prompt_eval_duration, eval_count, and eval_duration, all in nanoseconds. That is enough to compute the three numbers that matter: load time, prefill speed, and decode speed.

# bench.py
import json
import sys
import urllib.request

model = sys.argv[1] if len(sys.argv) > 1 else "qwen3-agent"
payload = {
    "model": model,
    "prompt": "Explain the CAP theorem in three sentences.",
    "stream": False,
}
req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
d = json.load(urllib.request.urlopen(req))

print(f"load     {d['load_duration'] / 1e9:6.2f} s")
print(f"prefill  {d['prompt_eval_count'] / (d['prompt_eval_duration'] / 1e9):6.1f} tok/s")
print(f"decode   {d['eval_count'] / (d['eval_duration'] / 1e9):6.1f} tok/s")
print(f"total    {d['total_duration'] / 1e9:6.2f} s")

Run it, change one variable, run it again. Decode speed is what you feel while reading a response. Prefill speed is what your agent feels when it stuffs 20k tokens of tool output into the prompt.

To find your practical ceiling, send a long prompt and watch prefill:

# long_context.py
import json
import urllib.request

filler = "Latency is a distribution, not a number. " * 3000
payload = {
    "model": "qwen3-agent",
    "prompt": filler + "

In one sentence: what is the main idea above?",
    "stream": False,
}
req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
d = json.load(urllib.request.urlopen(req))
print(d["prompt_eval_count"], "prompt tokens")
print(round(d["prompt_eval_count"] / (d["prompt_eval_duration"] / 1e9), 1), "prefill tok/s")

Keep the filler under your num_ctx. When prefill collapses at 24k while the model card advertises 128k, 24k is your real ceiling on that card, and no amount of prompt engineering changes it.

Troubleshooting

  • Context still 4096 after the change: the service was not restarted, or the variable went into the unit file instead of the drop-in. Check systemctl show ollama -p Environment.
  • PROCESSOR shows a CPU/GPU split: the weights plus the cache do not fit. Lower num_ctx, switch the cache to q8_0, or use a smaller quantization.
  • Replies stop mid-sentence: that is num_predict, not context. Compare eval_count with the limit you set.
  • The API returns 503: the queue is full. Add parallelism only if memory allows, otherwise serialize the work.
  • Quality drops after switching to q4_0: expected on some models. Go back to q8_0 and keep the context instead.

The loop worth keeping

Read context_length from /api/ps. Pin the number in a Modelfile for each job. Quantize the cache before you buy hardware. Keep the model warm on machines that can afford it. Then measure load, prefill, and decode after every single change.

Three numbers and one config file per model beats guessing why the agent forgot the third instruction again.

References

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation