Prompt caching across Claude, GPT and Gemini

9 min readSimon BudziakBy Simon Budziak

On this page10 sections
Title card reading Prompt caching, read at a tenth. A stylized navy stack of identical document panels, the lower ones dimmed and compressed into a single stored block wrapped in a gold bracket, with one bright new panel above it and a thin gold thread running from both out to a small terminal window.

We built a document analysis pipeline where the agent carried a 600-token system prompt and 12,000 to 15,000 tokens of contract text on every request, then answered ten questions per document. By question three we were paying for the same 15,000 tokens three times. The document had not changed. Only the question had. Prompt caching fixes exactly this, and the pricing is public: a cache read costs one tenth of a fresh input token at every major provider. This post covers what caches and what does not, the numbers each provider actually publishes, and the traps that quietly turn caching into a cost increase.

What changed since this was first written. Anthropic’s prompt caching is no longer beta and needs no beta header. OpenAI’s newer models gained explicit cache breakpoints and started charging for cache writes. The old framing of “Anthropic is explicit, everyone else is implicit” no longer describes reality, so the comparison below is rebuilt against current documentation rather than carried over.

Why static context is the expensive part

A document analysis request has three layers, and only one of them changes.

  • System prompt. Role, output format, constraints. Several hundred tokens, identical on every call.
  • Document context. The file under analysis. Ten to fifteen thousand tokens, identical across every question about that document.
  • User message. The actual question. Fifty to a hundred and fifty tokens, unique each turn.

Ten questions against a 15,000-token document bills roughly 156,000 tokens, and about 1,500 of those are the questions. The rest is the same prefix, repriced ten times.

Prompt caching keeps a server-side representation of a prompt prefix. Send the same prefix again inside the cache window and the provider skips reprocessing it, charging a fraction of the normal input rate. The question is still processed fresh. The scaffolding is what stops being repriced.

The economics providers actually publish

This is the part worth memorising, because the two leading providers have converged on the same multipliers.

Anthropic publishes them plainly: five-minute cache writes cost 1.25 times the base input rate, one-hour writes cost 2 times, and cache reads cost 0.1 times (Anthropic, “Prompt caching”).

OpenAI now matches on the newer families: cached input is billed at 0.1x the uncached rate and tokens written to the cache at 1.25x, and their docs are careful that the write rate “is the total rate for written tokens. It is not an additional charge on top of another full input-token charge” (OpenAI, “Prompt caching”). On models before that generation, cache writes carry no fee at all.

So the trade is the same wherever you run it: you pay a quarter more once to write, then a tenth as much every time you read. Two reads inside the window and the write has paid for itself. That is the whole business case, and it is also why a workload that never gets a second read is a workload caching makes more expensive.

A navy comparison card titled What a cached token costs. Three bars against a baseline of one times the uncached input rate: cache read at 0.1 times, a five-minute cache write at 1.25 times, and a one-hour cache write at 2 times. A note underneath reads that two reads inside the window repay a five-minute write.
Write once at a premium, read many times at a tenth. The break-even is the second read.

What each provider requires of you

The mechanisms differ more than the pricing does.

Anthropic is explicit and always has been. You mark content blocks with cache_control and get up to four cache breakpoints per request. It is no longer beta: there is no anthropic-beta header to set, which is the single most common piece of stale advice still circulating about it.

OpenAI is now both. Caching is automatic for eligible prompts, and the newer families added explicit breakpoints via prompt_cache_breakpoint, plus a prompt_cache_options.mode you can set to explicit to stop a changing suffix being written at all. The catch worth knowing: on those models prompt_cache_key is not merely a hint, it is what enables the more reliable matching, and OpenAI advises keeping each key under roughly 15 requests per minute or hits start dropping.

Gemini is implicit by default on 2.5 and newer models, with explicit cached content available through generateContent. Cache hits appear in usage.total_cached_tokens (Google, “Context caching”).

The practical read: implicit caching is a discount you may receive, explicit caching is a discount you designed for. If the saving matters to your unit economics, mark the breakpoint yourself rather than hoping the prefix matched.

The minimum nobody checks

Every provider has a floor below which nothing caches, it varies by model, and Anthropic is explicit that a short prompt fails silently: requests below the minimum “will be processed without caching, and no error is returned.”

This Is Fine meme. The dog sits in a burning room under the caption Prompt is under the cache minimum, and in the second panel says This is fine, captioned No error is returned.
The failure mode is silence. You are billed at full rate and nothing tells you.

The floors are not one number to memorise. Anthropic’s range from 512 tokens on its newest models up to 4,096 on some others; OpenAI’s newer families use a strict 1,024, while earlier ones vary from 1,024 to 2,048; Gemini’s run from 2,048 to 4,096. The only safe move is to check the table for the model you actually deploy, because the ordering is not intuitive: a newer, larger model can have a lower floor than the mid-tier model beside it.

Time is measured from the request, not the reply

Anthropic’s default cache lifetime is five minutes, with a one-hour option, and every hit refreshes it for free. The subtle part is where the clock starts. Their documentation: the lifetime “is measured from the start of the request that writes or reads the cache entry, not from the end of its response.” Their own worked example is the one to remember, because a streaming agent hits it constantly: if a response takes four minutes to stream, a follow-up that wants the same cached prefix has about one minute to start.

OpenAI’s newer families use a 30-minute lifetime that also refreshes on reuse, and several models support extended retention up to 24 hours through prompt_cache_retention. A long-gap workload is worth checking against that setting before concluding caching cannot help.

Wiring it up in LangChain

The current LangChain pattern for Anthropic is a content block carrying cache_control, with no headers and no special model configuration:

from langchain.chat_models import init_chat_model
from langchain.messages import SystemMessage

model = init_chat_model("claude-sonnet-4-6")

system = SystemMessage(content=[
    {"type": "text", "text": SYSTEM_PROMPT},
    {
        "type": "text",
        "text": f"Contract under review:\n\n{document_text}",
        # everything up to and including this block is cached
        "cache_control": {"type": "ephemeral"},
    },
])

Then the question goes in a separate, uncached message, so the prefix stays byte-identical across the session while only the suffix moves.

For an agent rather than a bare model call, LangChain now has a cleaner place to put this. Middleware can attach the cached block on every model request, so the caching decision lives in one wrapper instead of at every call site:

from langchain.agents.middleware import wrap_model_call
from langchain.messages import SystemMessage


@wrap_model_call
def add_cached_context(request, handler):
    blocks = list(request.system_message.content_blocks) + [{
        "type": "text",
        "text": f"<document>{document_text}</document>",
        "cache_control": {"type": "ephemeral"},
    }]
    msg = SystemMessage(content=blocks)
    return handler(request.override(system_message=msg))

Use content_blocks rather than content, as the LangChain docs specify, so the existing structure survives whether the original was a string or a list.

Verify it, because silence is the failure mode

Nothing errors when caching does not happen. You simply pay full price, so the metadata is the only evidence. LangChain now normalises this across providers in usage_metadata, which is a real improvement on reading each provider’s raw shape:

usage = response.usage_metadata or {}
details = usage.get("input_token_details", {})
print(details.get("cache_read", 0),
      details.get("cache_creation", 0))

Anthropic’s raw fields are cache_creation_input_tokens and cache_read_input_tokens; OpenAI reports cached_tokens and, on the newer families, cache_write_tokens; Gemini reports usage.total_cached_tokens.

The number that matters is the ratio between writes and reads, not either one alone. Writes high and reads low means you are paying the 1.25x premium repeatedly and collecting the 0.1x discount rarely, which is worse than not caching. OpenAI names the usual culprit: a timestamp or other request-specific content sitting before the breakpoint, so the prefix differs every time.

What to cache, and what will quietly break it

Cache the system prompt, the document or knowledge-base context for a session, tool definitions when you register many, and few-shot examples that do not vary.

Do not cache the user message, and be careful with conversation history: it grows, and summarising or compacting it mid-run rewrites the prefix. OpenAI is direct that truncation and compaction “can reduce prompt size, but they can also reset the reusable prefix”, which is a genuine tension with the summarization triggers we wrote about in LangChain model profiles for context management. Compaction saves tokens on this call and can cost you the cache on the next one.

The silent killers are all the same shape: anything dynamic ahead of the breakpoint. A timestamp, a user ID, a session counter, a reordered tool definition. Tool schemas and their ordering are part of the prefix, so a dictionary that serialises in a different order is enough to miss. An edited system prompt belongs on the same list, which is one practical argument for pinning prompts to immutable commits instead of editing them in place.

The takeaway

Prompt caching needs no model change, no architectural rewrite and no quality tradeoff, which makes it the rare cost lever with no real downside except the one nobody checks for.

  • Write once, read at a tenth. 1.25x to write, 0.1x to read. Break-even is the second read inside the window.
  • Explicit beats hopeful. Implicit caching is a discount you might receive; a breakpoint is one you designed for.
  • The floor is per model and fails silently. Between 512 and 4,096 tokens depending on what you deploy, with no error when you miss it.
  • Watch the write-to-read ratio. High writes with low reads is caching costing you money.

That is how we keep agentic pipelines affordable at Soba Labs: measure the ratio, put the breakpoint after the last thing that never changes, and treat a cache that never gets read as a bug rather than a rounding error.

Sources

Last updated:

How we build on LangChain in production