How to cut LLM spend without making the product worse

Cut LLM cost in order of risk to output, not in order of headline savings. Measurement, context trimming, and prompt caching change the price of a request without changing the task the model performs. Loop caps, task-class routing, and a model downgrade change behavior, so each one ships behind an eval suite that already passes on the model you run today.

Why the bill triples before anyone notices

Spend grows faster than usage because the payload per request grows at the same time. An agent resends the entire transcript on every turn, so a twelve-turn conversation pays for the first turn twelve times. Tool definitions ride along on each of those requests whether or not the model ever calls them, and retrieval that was tuned for recall keeps stuffing whole documents into a prompt that only needed two paragraphs.

The proposal that follows a bad invoice is almost always the same one. Somebody suggests moving every call to a smaller model, because the per-token rate is the one number visible on a pricing page. Nobody can say what that does to quality until a customer reports a worse answer, which is a slow and expensive feedback loop for a change that touches every request in the product.

How to reduce LLM API costs without hurting quality

There are six levers, ordered here by savings per unit of risk. The first three reduce billed tokens while leaving the model's job identical. The last three change what the model can do or how long it can run, so each carries a gate that has to pass before it ships.

1. Measure before you cut

Provider dashboards report spend by API key and model, which tells you nothing about which feature to fix. Attribute cost at the level you make decisions: per execution, per conversation, per tenant, per feature. Break each one into cached input tokens, uncached input tokens, and output tokens, because the three respond to completely different levers.

Then sort executions by cost, descending, and read the top of that list by hand. Cost distributions in agent products tend to be skewed, and a handful of runaway conversations usually explains more of the bill than the median request does. The method for building that per-conversation number, including how to attribute retrieval and tool calls, is in how to calculate cost per AI conversation.

2. Cut context before you cut models

Every token you send is billed on every turn that resends it, so input pruning compounds in a way output pruning does not. Four places usually hold slack:

• Retrieval scope. A top-k of 20 with 2,000-token chunks is 40,000 input tokens on every request. Lower k, shorten chunks, and add a relevance floor so weak matches are dropped instead of padded in.
• Tool definitions. Names, descriptions, and JSON schemas for every attached tool are serialized into each request. An agent carrying 40 tools that only ever calls 4 pays for the other 36 on every turn.
• Transcript history. Keep the last N turns verbatim and replace older turns with a short running summary, rather than resending the full history forever.
• Dead instructions. System prompts accumulate rules for edge cases that were fixed elsewhere months ago. Delete them and see whether anything moves.

The gate here is cheap: run your existing question set and confirm that retrieval still surfaces the passages the answers depend on. Recall is the thing a context trim actually threatens, and it is measurable without a model in the loop.

3. Cache the prefix you already send

Prompt caching stores a stable prefix at the provider and serves it back at a reduced input rate on later requests. It applies naturally to agent loops, conversations with history, and batch runs over one prompt version, because all three resend an identical opening block many times.

The mechanism is byte-exact prefix matching, and that single property explains every way caching fails to pay off:

• One changing byte near the top of the system prompt invalidates everything after it. A timestamp, a request id, or a tenant name in the opening block gives each request a private prefix and no reads.
• Cache writes are billed at a premium over ordinary input on some providers, so a prefix that is written and never re-read makes the bill worse, not better.
• Cached prefixes expire on an idle window that differs by provider. As of September 2026 an Anthropic entry lasts five minutes by default, with a one-hour option at double the write price, while OpenAI holds one for at least 30 minutes on GPT-5.6 and later. Reading an entry restarts its clock on both, so low-traffic tenants may still never hit a warm cache.

Put stable content first: tool definitions, then the system prompt, then per-request variables. Move rendered timestamps, execution ids, and tenant identifiers into the user message. The failure modes and the read-rate math are covered in how prompt caching actually saves money.

4. Cap output, turns, and tool calls

Unbounded loops are a cost bug that presents as a quality problem. Set explicit ceilings on generated tokens, agent turns, and tool calls per turn, and treat hitting one as an incident to investigate rather than a limit to raise:

{
  "model": "claude-sonnet-5",
  "maxTokens": 2048,
  "maxToolCalls": 10,
  "maxTurns": 12
}

One trap: on a reasoning model the output cap covers thinking tokens as well as visible text. A cap under roughly 1,024 tokens can be consumed entirely by reasoning, and the request returns empty content with a length stop reason, which reads as a broken feature rather than a saved dollar. Cap output generously on reasoning models and get your savings from turn limits instead.

The quality gate for this step is a replay of your longest recorded conversations against the new ceilings, checking that none of them now terminate before producing an answer.

5. Route by task class

Not every request needs the same model. Classify by what the call has to do, then pin a model per class, which is the practical shape of routing requests across models by task class:

Route on signals you already have, such as the surface the request arrived on, the flow step, or the tool being invoked. A model call that decides which model to call adds a request and a failure mode to every turn, and it is worth that cost only when the classes genuinely cannot be told apart from the request itself. When routing to a cheaper model pays off walks through where the break-even sits.

6. Downgrade the model last, behind the eval suite

By the time you get here the remaining spend is concentrated in calls that were doing real work. Treat a downgrade the way you would treat any behavior change: freeze an input set, run both variants over it, and compare pass rate and cost side by side before anyone approves it.

Build that input set from production rather than from imagination. Promote real executions into cases, including the ones that already failed, and keep every past regression in the suite permanently. Judge with a rubric on a fixed scale so two runs are comparable, and have a human review a sample of the scores, because an automated judge that drifts turns a downgrade into a rubber stamp.

The order in one table

What changes when the agent is customer-facing

An internal tool can absorb a bad week. A feature running inside your product for many tenants cannot, and aggregate eval numbers actively hide the damage: a suite that improves overall can still regress the one customer whose documents are longest, whose language is not English, or whose workflow depends on the tool call you just made unreachable behind a lower turn cap. Slice results by tenant before you approve anything, and weight the slices by revenue rather than by request volume.

Cost changes also move unit economics per plan tier, not just the total. A downgrade that saves money on the median account can still be net negative on the tier where the heaviest users live, and the reverse happens too. Per-customer usage ceilings belong in the same conversation, since a cap that protects margin on one plan is a broken product on another. Setting AI usage limits per customer covers where to put those ceilings.

Where this gets easier

Runtype runs the same eval suite across model and configuration variants and reports quality and cost together, so a proposed downgrade arrives with a pass rate and a per-execution cost beside each other instead of one number and a guess. Cases can be promoted from recorded production executions, including failures, and compared run to run and record to record, and cost is reported per execution, per record, and per batch, with cached and uncached tokens broken out so a caching change shows up as a number rather than an impression.