← Back to Blog
Oct 03, 2026 · 11 min read

Prompt Caching in Production: Cut LLM Input Costs by Up to 90%

AI AgentsLLMCost OptimizationPythonTutorial

If your app sends the same long system prompt or the same retrieved document on every request, you are paying the model to read that prefix from scratch each time. A support bot that fires 10,000 requests a day against a 15,000-token instruction block recomputes 150 million tokens daily for text that never changed.

Prompt caching fixes that. On Anthropic it is opt-in through a marker you place in the request. On OpenAI it runs automatically. On Gemini it comes in an automatic and a manual flavor. This guide walks through the Anthropic marker, how to confirm a cache actually hits, and how to pick a cache lifetime that matches your traffic. The commands are copy-pasteable and the pricing multipliers come from the providers' own docs.

Prerequisites

  • Python 3.10 or newer
  • The Anthropic SDK (pip install anthropic)
  • An Anthropic API key in ANTHROPIC_API_KEY
  • A prompt with a long, stable prefix. It must clear the model's minimum cacheable length, which ranges from 512 to 4,096 tokens depending on the model. Below that, nothing caches and you get no error.

Step 1: The no-cache baseline

Send the same prompt twice and watch the usage block.

import anthropic

client = anthropic.Anthropic()
SYSTEM = open("system_prompt.md").read()  # long and stable

for _ in range(2):
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=256,
        system=SYSTEM,
        messages=[{"role": "user", "content": "Summarize section 4."}],
    )
    print(resp.usage)

Both calls bill the whole system prompt at the base input rate. Nothing is cached, because you never marked anything as cacheable.

Step 2: Add a cache breakpoint

Wrap the system prompt in a content block and attach cache_control. Everything up to and including the marked block becomes the cached prefix.

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=256,
    system=[
        {
            "type": "text",
            "text": SYSTEM,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Summarize section 4."}],
)
print(resp.usage)

The first request pays a write premium. Every request after it, inside the cache lifetime, pays a read price.

Step 3: Verify the cache actually hits

A cache miss never throws an error. It silently costs full price. The usage block is the only place you can see the truth, and it splits input into three counters:

  • input_tokens: the total input count
  • cache_creation_input_tokens: tokens written to the cache, billed at 1.25x base for the 5-minute lifetime or 2x base for the 1-hour lifetime
  • cache_read_input_tokens: tokens served from cache, billed at 0.1x base

The cost of a request, in units of the base input price, works out to:

u = resp.usage
uncached = u.input_tokens - u.cache_read_input_tokens - u.cache_creation_input_tokens
units = uncached * 1.0 + u.cache_creation_input_tokens * 1.25 + u.cache_read_input_tokens * 0.10
print(f"{units:.1f} base-input units")

On a working cache you see cache_creation_input_tokens on the first call and cache_read_input_tokens on the next. If both stay at zero across repeated identical requests, your prefix is changing somewhere, and Step 4 is why.

Step 4: Never put changing content inside the cached block

The cache is a prefix match. It hashes the request in the order tools, then system, then messages, up to your breakpoint. One byte of difference anywhere before the mark produces a different hash and a full-price write.

The classic mistake is a timestamp or a request id stitched into the system prompt:

# Wrong. The timestamp changes every request, so the prefix never matches.
system=[{
    "type": "text",
    "text": f"Current time: {now()}\\n{SYSTEM}",
    "cache_control": {"type": "ephemeral"},
}]

Move the breakpoint to the last block that stays identical across the requests you want to share a cache:

# Right. Cache the stable block, keep the changing value outside it.
system=[
    {"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}},
    {"type": "text", "text": f"Current time: {now()}"},  # not cached
]

You can place up to four breakpoints when different sections change at different speeds. Tool definitions rarely change, but retrieved context might refresh daily, so give the tools their own breakpoint and leave the daily context outside it.

Step 5: Match the cache lifetime to your traffic

The default {"type": "ephemeral"} lives for five minutes and refreshes on every hit. The alternative, "ttl": "1h", lives for an hour at a doubled write premium.

A 5-minute write is 1.25x base and a read is 0.1x, so it pays for itself after a single read. A 1-hour write is 2x base, so it needs about two reads before it beats no caching at all. Pick by how often the same prefix gets hit:

Traffic to one prefix Lifetime
Several hits per minute 5-minute default
A hit every few minutes 5-minute default
A hit every 10-30 minutes 1-hour
A hit every hour or rarer Probably not worth caching

One trap: the lifetime is measured from the start of the request, not the end. If a response takes four minutes to stream, the next request using the same prefix has to arrive about a minute later to hit a 5-minute cache.

Step 6: For chat, let the API manage the breakpoint

In a multi-turn conversation the history grows every turn. Instead of moving your marker by hand, drop a single top-level cache_control and let the API push the breakpoint to the last cacheable block.

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=256,
    cache_control={"type": "ephemeral"},  # top-level: automatic caching
    system="You are a support agent for Acme. Follow these guidelines...",
    messages=history,  # grows each turn
)

One limit matters here. The API looks back at most 20 content blocks from a breakpoint to find a prior cache write. A turn that appends more than 20 tool-use and tool-result blocks can miss the previous turn's entry and re-pay the write. Add a second, closer breakpoint if your agent turns are that heavy.

Step 7: OpenAI and Gemini

OpenAI caching is automatic and free. Any prompt of 1,024 tokens or more gets the discount with no code change, and the docs claim up to 80% lower latency and up to 90% lower input cost. You confirm it by reading usage.prompt_tokens_details.cached_tokens. The optional prompt_cache_key parameter helps route requests that share a long prefix to the same machine.

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "Summarize section 4."},
    ],
)
print(resp.usage.prompt_tokens_details.cached_tokens)

Gemini has both modes. Implicit caching is on by default for 2.5 and newer models and is best-effort, so it gives no guarantee. Explicit caching is a manual step but guarantees the discount. Create a cache once, then reference it. The default TTL is one hour, and the minimum is 2,048 tokens on the 2.5 family and 4,096 on the 3.x family.

from google import genai
from google.genai import types

client = genai.Client()

cache = client.caches.create(
    model="gemini-2.5-flash",
    config=types.CreateCachedContentConfig(
        contents=[types.Content(role="user", parts=[types.Part(text=SYSTEM)])],
        ttl="3600s",
    ),
)

resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Summarize the key points in three bullets",
    config=types.GenerateContentConfig(cached_content=cache.name),
)
print(resp.usage_metadata.cached_content_token_count)

When caching pays off, and when it does not

Situation Cache? Why
Long stable system prompt, high request volume Yes One write, then 0.1x reads
RAG with a fixed document set hit many times Yes The retrieved prefix dominates the cost
Prompt shorter than the model minimum No The prefix silently does not cache
Content that changes on every request No The hash never matches, so you only pay write premiums
Traffic so sparse the TTL expires before the next hit No You pay 1.25x every request and never read

Conclusion

Prompt caching turns a stable prefix into a one-time cost. The order that matters: put stable content first, mark the last identical block as the breakpoint, pick a TTL that matches your real traffic, then read the usage counters to confirm hits. A cache that never hits is worse than no cache, because you pay the write premium every time and call it a saving.

Next step: log cache_read_input_tokens as a hit-rate metric, and re-run the cost formula above on a week of real traffic before you consider the feature done.

References

Need Help Implementing This?

I help teams design and build scalable cloud infrastructure, DevOps pipelines, and production-grade systems.

Book a Free Consultation