Prompt caching: what it saves, and where it quietly breaks

A provider caches a prefix of your request, counted in tokens from the very first one. A later request reads that prefix from cache only when every byte before the cached point matches. Anything variable near the front, a timestamp, a customer name, a tool list that serialized in a different order, moves the boundary and the discount disappears.

The failure is quiet because nothing errors. Requests succeed, latency looks ordinary, and the bill rises, because several providers charge a premium to write a cache entry. A team that switched caching on in April discovers in June that 4 percent of input tokens came from cache and the rest paid that premium.

Two things make it hard to catch. Cache counters arrive inside the usage object of each response, in fields most logging pipelines drop before anything reaches a dashboard. And whatever broke the prefix is usually a line someone added for a good reason, such as the current date or the account name.

How does prompt caching work and when does it not help

A model processes input left to right and builds internal state as it goes. Prefix caching stores that state for a run of tokens and reuses it when a later request begins with the same tokens. The unit is a prefix, so a segment in the middle of your prompt cannot be cached on its own, no matter how stable it is.

Matching is exact and starts at position zero, so a tiny edit far into a long prompt still costs you everything after it. Providers also impose a minimum length below which nothing is cached. As of September 2026 that floor is 1,024 tokens on Claude Sonnet 5 and OpenAI's GPT-5.6 and later, 512 on Claude Opus 5, and 4,096 on Claude Haiku 4.5 and the Gemini 3.x Flash models. The smaller model you moved a step onto is often the one that quietly stops caching.

Request order decides what sits at the front. Anthropic documents the prefix as tools, then system, then messages; OpenAI renders its own hidden system content, then tools, then your developer message, then the conversation. On both, the tool list is compared before your system prompt. Google publishes no ordering for Gemini and advises only that large shared content go early.

Caching does not help at all in four situations: a one-shot request whose prefix is never sent again, a prompt below the minimum length, a workload whose long content is per-request retrieval rather than shared context, and traffic spread thinner than the cache lifetime, where every request is the first one.

What silently breaks the prefix

The third and fourth rows are the ones teams argue about. A tool description is part of the cached prefix, so editing one word in one tool invalidates the cache for every request that carries the full tool list, including tools nobody touched. Batch tool-definition edits and ship them together rather than trickling them across a week.

Explicit and automatic caching are different contracts

The split decides how much control you have and what a mistake costs.

Explicit caching asks you to mark where the cacheable prefix ends. Anthropic uses cache_control breakpoints on content blocks, placed at the end of tools, at the end of the system prompt, or after a stable stretch of conversation. You get precision, and you also own the mistake: a breakpoint placed after a volatile line caches nothing useful and still charges the write premium.

Automatic caching matches repeated prefixes with no parameter. Gemini's implicit caching is on by default for 2.5 and newer models and needs no cached-content object, and DeepSeek caches prefixes to disk the same way. OpenAI does both: implicit by default, plus explicit breakpoints on GPT-5.6 and later. Nothing to configure means nothing to misconfigure, and no way to say which 30,000-token block is worth keeping.

The same prompt restructuring helps under both contracts. Stable material first, per-request material last: the automatic match runs longer and the explicit breakpoint becomes worth setting.

What cache reads and writes cost, and how long they last

Three numbers describe the economics: the base input rate, the write rate, and the read rate. As of September 2026 Anthropic and OpenAI both charge 1.25x base input to write a short-lived entry and 0.1x to read one, and Anthropic's one-hour entries cost 2x to write. Google and DeepSeek charge no write premium: Gemini cached input is a tenth of base input plus hourly storage on explicit caches, and a DeepSeek cache hit bills about a thirtieth of a miss.

The shape matters more than the exact figures. One warm prefix at those rates pays for itself after roughly one and a third requests, which is why caching looks free in a demo where you send the same prompt five times in a row.

Now apply the same rates to a 4 percent hit rate. Of every 100 requests, 96 write and 4 read, so input cost lands near 1.20x what you would have paid with caching switched off. That is the whole failure: caching that misses is more expensive than no caching, and the effect is invisible unless someone is reading cache counters. The wider version of this trap, an optimization that raises spend while looking like a saving, is covered in reducing LLM costs without breaking the product.

Lifetime is short, and every provider here restarts the clock on a read. An Anthropic entry lives five minutes from the start of the request that touches it, with a one-hour option at the 2x write price. OpenAI holds one at least 30 minutes past its last write or read on GPT-5.6 and later, and five to ten idle minutes on older models. A busy conversation stays warm indefinitely; an idle one falls out between turns.

The order to put things in

Sort every piece of context by how often it changes, then lay the prompt out in that order.

Rows 5 and 6 belong in the user turn. Moving the date out of a system prompt feels wrong the first time, and the model reads it just as well from the user message, where it costs one uncached tail instead of one uncached everything.

Row 1 needs a deliberate sort. Serialize tool definitions from an ordered list with sorted object keys rather than from whatever your registry iterates, or two replicas of the same service will build byte-different prompts and neither will ever read the other's cache entry. A larger tool pool also makes the prefix longer and its invalidation more expensive, which is a reason to keep the pool tight.

The multi-tenant trap

Putting the tenant name at the top of the system prompt gives every customer their own cache entry. With ten heavy tenants that is fine. With 500 tenants each sending a few requests an hour, every entry expires before it is read twice, and you pay the write premium on nearly every request while a dashboard reports that caching is enabled.

The fix people reach for next is worse. Hoisting a tenant's documents or configuration into the shared prefix to make it long and stable produces one cached prefix that holds one customer's material and serves requests for other customers. Anthropic and OpenAI both isolate cache entries per organization, and Anthropic isolates per workspace inside one, so nothing crosses an organization boundary. Inside your own, nothing partitions entries by your end customer, so that boundary is yours alone to hold.

Keep the shared prefix to material every tenant may see, and let tenant content sit after it. A tenant with enough traffic to keep its own longer prefix warm can have one; a tenant with three requests a day reads the shared prefix and takes the miss on its own segment. Which resources carry tenant scope is the checklist in tenant isolation for AI features.

Measuring hit rate instead of assuming it

Every provider reports cache activity in the usage object, under different names. Capture these fields on every call, alongside the prompt version and the tenant, before anything is aggregated:

{
  "anthropic": {
    "usage": {
      "input_tokens": 412,
      "cache_creation_input_tokens": 0,
      "cache_read_input_tokens": 24180
    }
  },
  "openai": {
    "usage": {
      "input_tokens": 24592,
      "input_tokens_details": { "cached_tokens": 24064, "cache_write_tokens": 0 }
    }
  },
  "google": {
    "usage": {
      "total_input_tokens": 24592,
      "total_cached_tokens": 24064
    }
  },
  "deepseek": {
    "usage": {
      "prompt_cache_hit_tokens": 24064,
      "prompt_cache_miss_tokens": 528
    }
  }
}

Compute the rate over tokens, not over requests. A request-weighted figure counts a call that matched 200 tokens the same as one that matched 24,000, which is how a 4 percent saving gets reported as an 80 percent hit rate. The token-weighted version is cache read tokens divided by all input tokens, including the ones written and the ones never cached.

Segment the result by prompt version. An aggregate number blends the version you fixed last Tuesday with the one still carrying a timestamp, and the average moves too slowly to tell you which change worked. The same segmentation drives the cost per AI conversation figure, since cached and uncached input have different rates and a blended per-conversation cost hides both. Which model each step runs on changes the arithmetic again, under model routing.

Two checks are worth automating. Alert when a version's token-weighted rate drops below what you measured the day you shipped it, since that is the signal a new line entered the prefix. And re-measure after any deploy touching tool definitions, the change most likely to reset the cache and least likely to look related.

Where this gets easier

Runtype reports cached and uncached tokens per execution, so hit rate is a number you read rather than one you infer from a bill, and a prompt restructure can be checked against the run that followed it. Cost is recorded per execution, per record and per batch with the cached portion separated, and a flow that puts a per-run variable such as {{_now}} or {{_execution.*}} in a system prompt gets a non-blocking validation recommendation at save time, before the prefix ships. Details of that behavior are at prompt caching.