When to route to a cheaper model, and how to know it is safe

Route on task class, not on a guess about how hard a request looks. Downgrade one step at a time, pick the step whose output something downstream already checks, and prove the change by running the same eval cases against both models. Write the rollback before the change ships.

A product making several model calls per request usually has one or two carrying most of the bill and one or two carrying most of the risk. They are rarely the same call. Separating them is the whole exercise, and then moving exactly one.

Should I use a cheaper model for some requests

For requests whose class you know before the call is made, yes. A classification step, an extraction step with a schema, a summary written into a record that nobody reads in real time: each has a stable job, a checkable output, and a failure that surfaces in the next step rather than in a customer's inbox six weeks later.

The requests worth leaving alone are the ones where you would have to guess. Guessing is where teams reach for a difficulty router, a classifier or small model that reads each request and picks a tier. It runs on every request, including the ones headed for the expensive model anyway, so it is a fixed tax against a variable saving.

Do the arithmetic before the build: if the router costs R per request, the downgrade saves S on each request it moves, and it moves a share p of traffic, the gain is p × S − R. The ratio depends on the pair. Vercel's AI Gateway model list in September 2026 shows these input and output list prices per million tokens:

With a router at a tenth of the expensive call, p has to clear a fifth of traffic to break even on the Sonnet to Haiku pair and an eighth on a five-to-one pair, before latency and the new failure mode are counted.

The other property that makes a difficulty guess expensive is that its mistakes are quiet. A small model handed a request above its weight still answers fluently, in the right format, at the right length. Nothing errors, no trace turns red, and the only evidence is a support ticket that arrives after the routing rule has been live long enough that nobody connects the two.

The signals that route reliably

Five signals are worth routing on, and they share a property: each is known before the call is dispatched, costs nothing to evaluate, and appears in the run record afterwards so a bad rule can be traced back.

Input length is the most misused. It is a good proxy for cost and a poor one for difficulty, so treat it as a ceiling rather than a router: send anything over the threshold to the larger model, and do not send anything under it to the smaller one on that basis alone.

Pick the step before you pick the model

Run each candidate step through five checks. A step that passes all five is a safe first downgrade, and a step that fails two or more should wait until you have evidence from the first one.

1. It is a real share of spend. Cost attributed per execution and per feature, not a guess from the provider dashboard. A step at three percent of the bill is not worth the review it will cost you. The method is in how to calculate cost per AI conversation.
2. Its output is checked by something other than a person. A JSON schema, a parser, an enum of allowed labels, an assertion in the next step. The check turns a quality regression into a measurable failure rate.
3. Its failure surfaces immediately. A wrong intent label picks the wrong branch and the run visibly goes somewhere odd. A slightly worse stored summary surfaces months later, in a different feature, with no way to attribute it.
4. Nothing it produces reaches a customer verbatim. Drafting text a customer reads is the product. That step keeps the budget.
5. The change is one configuration value. If the downgrade requires a rewritten prompt to work on the smaller model, you are testing two changes at once and cannot attribute the result to either.

Often only one step passes all five on the first pass, usually a classifier or an extractor. Start there, ship it, and let the measured result decide whether the second candidate is worth the same effort. The broader ordering of cost levers, and why a model downgrade comes last among them, is in cutting LLM spend without making the product worse.

Evidence: two arms over the same cases

A routed arm and an unrouted arm is one experiment with a single variable. Freeze the system prompt, the tool definitions, the retrieval configuration, the case inputs, and the sampling parameters. Change the model id on one step. Run both arms over the same case set, built from recorded production executions rather than from invented examples.

Compare per case, never by average. A run that gains on the easy half and loses on the hard half reads as flat, and the hard half is the half that generates tickets. List the cases that moved from pass to fail and read them, because five explainable regressions you accept is a different decision from five you cannot explain.

Add the measures a judge will miss. First-attempt schema-valid rate, tool-call sequence equality against the baseline, median and p95 output tokens, and the escalation rate if you are testing the pattern below. A judge score can hold steady while the format-valid rate slips, and the parser downstream is what customers experience. The same discipline for provider version changes is in catching a model upgrade that quietly degrades your product, and the case-set and scoring mechanics sit with AI agent evals.

Then shadow before you serve. Run the cheap model in parallel on a sample of live traffic, discard its answer, and compare the two outputs on the mechanical measures. A shadow window costs the cheap model's price on the sampled share and buys you the one thing an eval set cannot give: the inputs your customers are sending this week, including the ones nobody thought to write a case for.

Escalate on a trigger a machine can see

The escalation pattern sends everything to the cheap model first and hands off to the expensive one when a defined trigger fires. It works when the trigger is mechanical:

{
  "step": "extract-order-fields",
  "primary": { "model": "claude-haiku-4-5", "maxTokens": 800 },
  "escalate": {
    "model": "claude-sonnet-5",
    "on": [
      { "trigger": "schema_invalid" },
      { "trigger": "empty_output" },
      { "trigger": "required_field_null", "fields": ["order_id", "customer_email"] },
      { "trigger": "self_reported_confidence_below", "field": "confidence", "value": 0.7 }
    ]
  }
}

Every trigger there is readable by a parser, though the confidence field is the weakest of the four, since a stated confidence is the model's own claim about itself and needs calibrating against your own cases before a threshold on it means anything. The moment a trigger requires a second model to judge the first model's answer, the pattern has turned back into a difficulty router with the tax moved downstream, and the arithmetic from the first section applies again.

The cost side is easier than it looks. Sending everything to the cheap model and escalating a share e costs C plus e × E against a baseline of E, so break-even is 1 − C/E. At the two-to-one Haiku 4.5 to Sonnet 5 ratio above, escalation can run up to half of all traffic before the pattern stops saving money; on a five-to-one pair such as GPT-5 mini to GPT-5 the ceiling is eighty percent. Latency is the constraint that binds first: an escalated request pays both calls end to end, so a p95 that was acceptable at one call may not survive two.

Cap the escalation depth at one hop. A chain that tries three models in sequence turns a transient provider problem into a triple-priced request and a timeout the customer sees, and it makes the run record hard to read afterwards.

The rollback you write first

A downgrade is one field, so write down the old value, the signal that restores it, and the person who owns the decision, before the change is deployed. Without a named signal, the rollback happens when someone notices, which is the slow feedback loop the eval arm was supposed to replace.

Four signals are usually enough: first-attempt schema-valid rate below the baseline you measured, escalation rate above the bound your arithmetic assumed, any per-case regression in the suite that runs on a schedule, and a rise in a specific support-ticket category. Each needs a threshold written as a number before the change ships, because a threshold chosen after the fact is chosen to justify whatever is currently happening.

Two record-keeping details make the rollback evaluable. The run record has to name the model that produced each answer, including when an escalation or a fallback moved the call, or a routing regression is indistinguishable from a prompt regression. And every surprise the downgrade produced becomes a permanent case in the suite, so the next person to propose the change inherits the evidence. The wider set of decisions, including fallback chains and cached-token reporting, is covered in model routing.

Where this gets easier

Model choice is per-step configuration in Runtype, so a proposed downgrade is a value on one step rather than a fork in the codebase, and the evidence comes from the eval suite that already guards the step: the same cases against both models, compared run to run and record to record, with cost per execution beside the score. Cases can be promoted from recorded executions, judge scores reviewed by hand on a sample, and each run record carries the model that produced the answer, so a rollback is the previous value of a field.