How to catch a model upgrade that quietly degrades your product

Treat the model as a variable and hold everything else fixed. Run one suite of real cases against the shipped model and the candidate, compare case by case instead of by average score, then shadow the candidate against live traffic before you cut over. Most of the damage from an upgrade lands in format, refusal and tool-selection behavior, and no public benchmark measures any of those.

New model version broke my prompts

The shape of this is familiar. A provider ships a model that is better on every published measure, someone swaps the id, and support tickets change character within a day. Answers get longer. A field that used to arrive as bare JSON now arrives wrapped in a fenced code block, so the parser downstream throws on one request in twenty.

Nobody can prove the model caused it, because the only evidence is a handful of screenshots and a feeling. The migration then stalls in the worst position: the old model is on a retirement schedule you do not control, and the new one is blocked on an argument nobody can settle with data. The way out is a comparison you can rerun, not a longer argument.

What degrades quietly

Four failure classes account for most upgrade regressions, and each has a mechanism you can measure directly rather than infer from a score.

• Format drift. The model adds a preamble, wraps JSON in a fence, renames a key from customer_id to customerId, or returns an array where it used to return a single object. Measure it as format-valid rate: the fraction of outputs that parse and match your schema on the first attempt, with no repair step.
• Refusal drift. Safety behavior is retuned between versions, so borderline requests that used to be answered now get a hedge or a decline. Measure it per intent category, not globally, because the movement concentrates in two or three categories (medical, legal, anything that reads like account access) and disappears in an average.
• Tool-selection drift. The model picks a search tool where it used to pick a direct lookup, calls the same tool repeatedly on an empty result, or stops calling a tool it used to call. Measure the exact tool-call sequence per case and diff it, since a turn that hits its tool-call budget ends without an answer even though nothing errored.
• Verbosity and cost shift. Output tokens per turn move, which changes cost per conversation, time to first useful sentence, and whether your chat UI truncates. Record median and p95 output tokens per case rather than a total, because a small number of runaway cases moves the bill more than a general drift.

Latency belongs in the same list, measured at p95 rather than mean. A model that is faster on short answers and slower on long ones can leave the average flat while your slowest ten percent of conversations get materially worse.

Step 1: know whether your model id is pinned or routed

There are two ways to name a model, and they fail differently. A pinned id names one immutable snapshot, so the same request returns the same behavior distribution for as long as the snapshot exists. A routed id names a family (claude-sonnet-5, gpt-5.6-terra) and resolves to whatever the provider currently serves for that family, which means it keeps working when a snapshot retires and also changes without a deploy on your side.

Neither is correct in every position. Pin in the eval harness, so a baseline stays a baseline and a diff means something. Route in production once a suite runs on a schedule against the routed id, so you find out the day behavior moves instead of the day a customer notices. The trade-offs between families and providers are covered in model routing.

Then write down the deprecation window. Providers publish a retirement date for a pinned snapshot and name the replacement, usually months ahead and usually on a changelog page rather than in an email to you. As of September 2026 the named replacement is a recommendation and nothing routes to it.

Anthropic's deprecation page states that requests to retired models will fail, OpenAI's says a shut-down model is no longer accessible, and Google says the same for retired Gemini ids. A request naming a retired snapshot returns an error, so a routed id is the only kind that survives a retirement on its own.

Put that date in the same tracker as your other dependency deadlines, because a migration you start two weeks before the cutoff is a migration you will do without a gate.

Step 2: build a harness that varies exactly one thing

A comparison run is only evidence if the model is the only difference. Freeze the system prompt, the tool definitions, the tool implementations, the case inputs, and every sampling parameter you set, then run the same suite twice with different model ids. Sampling parameters need care: a value one provider ignores may be honored by another, so record the exact request body you sent rather than trusting the config that produced it.

{
  "suite": "support-triage-v4",
  "hold_constant": ["system_prompt", "tools", "cases", "temperature", "max_output_tokens"],
  "runs_per_case": 3,
  "candidates": [
    { "label": "shipped", "model": "claude-sonnet-5" },
    { "label": "candidate", "model": "gpt-5.6-terra" }
  ],
  "record": ["output", "format_valid", "tool_calls", "output_tokens", "latency_ms", "refused"]
}

Run each case more than once. A single run per case cannot separate a real regression from ordinary sampling variance, and three runs is usually enough to see whether a case flipped or merely wobbled (why identical prompts return different answers). The suite itself is the same asset you use for prompt changes, described in regression testing prompt changes, and the case-selection method is the one in building an eval set from production.

Step 3: read wins and losses per case, never the aggregate

Two scores tell you almost nothing. A candidate that gains on one intent and loses on another lands within a point of the baseline, and the point it lost is the one your customers feel. Produce a table with one row per case and the delta on each measure you recorded, then sort by regression rather than by improvement.

The single number that matters most is the count of cases that moved from pass to fail. An upgrade that fixes nine cases and breaks two is still a decision, and it is a decision someone should make with the two named in front of them. Judge scoring helps with the cases where correctness is a matter of degree, and it needs human review of individual scores before you trust it as a gate, which is part of what a working AI agent eval practice covers. Tool sequences deserve their own assertions, described in testing agent tool calls.

Step 4: shadow the candidate on live traffic

A suite covers the cases you thought of. Production carries the ones you did not, so run the candidate against a sample of real requests without returning its output to anyone. Mirror the request, store both responses, and diff them offline on the same measures the suite tracks.

Three practical constraints. Sample rather than mirror everything, because a shadow run is a second inference call on every sampled request and doubles that share of your bill (routing work to cheaper models is a separate decision, and mixing it into an upgrade test confounds both). Never let a shadow response reach a user, a webhook, or a write path, which means the shadow run needs its own tool implementations or a read-only mode. Run it for at least one full weekly cycle, since traffic on a Monday morning looks nothing like traffic on a Saturday.

Compare the two response sets on distributions, not on individual pairs. Format-valid rate, refusal rate per intent, median and p95 output tokens, p95 latency, and the distribution of tool-call counts will each move or not, and a shift in any of them is a question to answer before the cutover rather than after it.

Step 5: write the rollout gate before you look at the results

A gate written after the numbers arrive is a rationalization. Write the thresholds first, in a form someone else can check, and treat any of them failing as a stop rather than a discussion. A workable set for a customer-facing agent looks like this.

1. Zero cases in the regression set move from pass to fail. Cases that fail on both models are tracked separately and do not block.
2. Format-valid rate on the candidate is greater than or equal to the shipped model's, measured on the shadow sample rather than the suite.
3. Refusal rate moves by no more than an agreed amount in every intent category, checked per category rather than overall.
4. p95 latency and median cost per conversation stay inside the budget the product already commits to.
5. A named person can roll back by changing one configuration value, without a deploy, and that path has been exercised at least once.

Then roll out by share of traffic, not by flipping a global default. Start at a small percentage, hold it long enough to see a full traffic cycle, and keep the shipped model reachable for the whole ramp. Pick the rollback trigger in advance, phrased as a metric and a threshold, so the person on call at 2am does not have to relitigate the decision.

Keep the regression cases afterward. Every case the candidate broke is a case worth running against the next upgrade, and a suite accumulated this way is the reason the third migration takes an afternoon instead of a quarter.

Where this gets easier

Runtype treats the model as configuration rather than code: a model config per agent or per step, changed without touching the prompt, the tools or the deployment. The same eval suite runs across model variants and returns a run-to-run and record-level comparison, so you read which cases moved rather than two aggregate numbers, with judge scores you can review individually, coverage reporting over the cases you have, and draft versions that let a candidate model run while the published version keeps serving customers. Traces carry per-step input and output, tool calls with their arguments and results, latency, and cost per execution with cached and uncached tokens separated, which is where the format, refusal and tool-selection classes above become visible without extra instrumentation. What eval suites cover is documented at what are evals.