Model routing and cost control per tenant

Route model calls by task class, not by a guessed difficulty score: choose a model per step, define what replaces it when it fails, report cached and uncached input tokens separately, and run the same eval suite before any cheaper model ships. That is model routing for a product team. It is the set of per-call decisions about which model serves a request and under what fallback, plus the record of what actually ran and what it cost the tenant it ran for.

The advice below assumes the agent is customer-facing and multi-tenant: many customers on a shared provider key, several entry points, and a price to defend. An internal assistant can get away with one model and a monthly bill.

Why model routing is hard

The difficulty classifier is a guess made by another model

Most routing products offer a router that scores each request for difficulty and sends the easy ones to a small model. The score comes from a classifier or a small model call, so it adds latency and cost to every request, and its mistakes are invisible: a small model's wrong answer is still fluent, so the dashboard records a success. Difficulty is also a property of the tenant, not the prompt. "Summarize this ticket" is easy for a five-line ticket and hard for a forty-turn escalation with three attachments.

A silent fallback changes the answer without changing the log

Provider incidents are routine, and a fallback that switches to another model keeps the request alive. The first version anyone builds does not record which model produced the reply, so when a customer reports that the assistant "started answering differently on Tuesday", every trace shows the configured model. Provider failover has the same shape from the other side: a gateway that moves a Claude call from Anthropic to Bedrock or Vertex AI during an outage has sent the prompt to a company that may not be in your data-processing terms.

Cached tokens are billed differently and reported the same

Anthropic, OpenAI and DeepSeek charge a cache read at a fraction of the input rate, and some charge a premium for a cache write. A usage report that sums input tokens overstates a well-cached workload and hides regressions in it. Put {{_now}} into a system prompt and every request after that misses the cache: the bill climbs, and the token count does not move. The signal is a cache-read count at zero, visible only if cache reads are a separate column.

Spend lands on the account, not the customer

A provider bill arrives per API key. When every tenant runs on the same key, the normal case for a SaaS product, cost per customer is reconstructed afterwards from whatever metadata each code path remembered to attach. Entry points multiply: web chat, a Slack surface, an API a customer's backend calls, a nightly job. One that forgets the tenant leaves spend in an empty bucket, and one tenant's retry loop (a tool returning an empty array the model reads as a failure) shows up as a spike on the whole account.

A cheaper model ships on a hunch

Downgrading a step from claude-sonnet-5 to claude-haiku-4-5 is a one-line change that halves the token price: Anthropic's pricing in September 2026 is $2 and $10 per million input and output tokens for Sonnet 5, against $1 and $5 for Haiku 4.5. What it did to quality requires the same cases run on both models with the same judge. Teams without that suite compare a few prompts by eye and learn the rest from support tickets.

How it works

Model routing has five parts, one per failure mode above.

Route by task class

A customer-facing agent is rarely one model call. It is a classifier, a retrieval step, a draft and a check, each a task class with a stable difficulty profile. Choosing the model per step when the flow is designed replaces the per-request guess with a decision someone can read.

The tradeoffs behind each row are worked through in routing to cheaper models. The table fixes where the decision lives: on the step, next to the prompt it serves.

Define the fallback, and make the run record it

A fallback chain is an ordered list of what to try when a call fails, plus the triggers that start the chain. The shape below is a flow prompt step; an agent carries the same chain on its model settings.

{
  "id": "draft-reply",
  "type": "prompt",
  "config": {
    "model": "claude-sonnet-5",
    "text": "Draft a reply to the customer's message: {{ticket.body}}",
    "outputVariable": "draft",
    "errorHandling": {
      "onError": "fallback",
      "triggers": [{ "type": "error" }, { "type": "empty-output" }, { "type": "slow", "afterMs": 8000 }],
      "fallbacks": [
        { "type": "retry" },
        { "type": "model", "model": "claude-haiku-4-5", "maxTokens": 600 },
        { "type": "message", "message": "I could not draft a reply just now. A teammate will follow up." }
      ]
    }
  }
}

The empty-output trigger exists because a provider can refuse without a transport error, and a retry with the same prompt refuses again, so the chain should advance to a different model or a fixed message. The slow trigger with afterMs turns a hung provider into a bounded wait rather than a timeout the customer sees. The record of the run then has to name the model that produced the answer, or the fallback is a silent behavior change.

Report cached and uncached tokens separately

Per call, keep four counts rather than two: input tokens, cache-read tokens, cache-write tokens, output tokens, each priced at its own rate. Aggregated per model, the useful figure is the share of input served from cache, because a drop in that share is the earliest sign that a prompt change broke the prefix.

The prefix rule is simple to state and easy to break: everything stable (system prompt, tool definitions, earlier turns) comes first and stays byte-identical, and every per-run value (time, execution id, the customer's message) goes into the user prompt or a later step. Which workloads get cache reads at all is covered in prompt caching.

Budget per customer in two layers

The first layer bounds the worst case per run so that metering can never be too late: a cap on tool calls per turn, a cap on turns, a cost ceiling on the run, and a maxTokens on each step. Those controls stop the empty-array retry loop at call ten instead of at the invoice. The second layer meters cost per execution with the tenant and end-user identity attached at execution time, and computes spend per customer against a plan allowance from those figures.

Only the second layer answers what a conversation costs for one customer. How to compute it is at cost per AI conversation, how to turn it into a price is at pricing AI features, and the enforcement side, including what to do when a customer crosses the allowance mid-conversation, is at usage limits per customer.

Gate every downgrade on the same eval suite

A model change is a release:

1. Keep an eval suite built from recorded production executions rather than imagined cases.
2. Run the suite on the current model for a baseline judge score, and review a sample of the scores by hand.
3. Run the same suite on the candidate model and compare run to run, at the record level, so you can read which cases regressed rather than an average.
4. Ship the downgrade only if the regressed cases are ones you accept, and add each surprising regression to the suite as a permanent case.

The suite is the same one that guards prompt changes, which is why it lives with AI agent evals rather than billing. A sequence for cutting spend without a quality surprise is at reducing LLM costs safely.

What changes when the agent is customer-facing and multi-tenant

Attribution has to be produced at execution time. An internal tool can tag spend in one place, because it has one entry point. A product has a web widget, a Slack app, an API, a scheduled job and an MCP server, and each is a place to forget the tenant. The runtime that executes the call is the only component that sees every entry point, so it is the only place identity on a cost figure can be guaranteed.

Fallback becomes a data-processing question. When the prompt contains a customer's data, the provider that receives it after a failover is a party to that customer's agreement. A team with a signed list of subprocessors needs the fallback chain to reflect that list, so a routed name a gateway may move between Anthropic, Bedrock and Vertex AI gets pinned on the steps where the path matters.

Budgets stop being a report. One tenant's loop on a shared key consumes the provider's rate limit for every other tenant, so a per-turn bound is a fairness control as much as a cost control. And a downgrade a suite approved on average may have regressed exactly the task class one customer segment relies on, which is why the record-level comparison matters more than the mean.

Where Runtype fits

Runtype holds the per-step model decision, the fallback chain and the cost record in one place.

Start by sending traces. An OpenTelemetry-instrumented agent exports OTLP/HTTP to https://api.runtype.com/v1/otel, and each run arrives with its trace tree, token usage and a display-only cost estimate, filed under the tenant and end user it ran for. That is cost per run and per tenant before any routing changes. Your provider still bills you, and two instrumentations at the endpoint double both figures.

Register the agent and Runtype calls it, recording tool calls and cost per run rather than importing them. An MCP surface points the other way, handing flows, agents, records and tools to your existing loop. Rebuild the one capability whose routing actually matters, once a harvested suite shows the cheaper model holds.

For steps Runtype executes, each prompt step and agent picks its model by a routed name such as claude-sonnet-5, or a provider-pinned name when the path must stay fixed, on your keys or the platform's (model routing and data flow). The fallback chain in the sample above is the native shape, and the run records the configured model, the resolved provider name when it differs, and the fallback that produced the answer. A platform-key call that fails transiently with no error handling gets an automatic recovery chain on another provider serving the same model; your own key does not.

Prompt caching is managed for the Anthropic, OpenAI and DeepSeek families, each prompt step reports cache-read and cache-write counts separately, and saving a flow with {{_now}} in a system prompt raises a CACHE_VOLATILE_SYSTEM_PROMPT recommendation (prompt caching).

Run bounds are agent settings: maxToolCalls per turn (default 10, at most 100), maxTurns between 1 and 100, an optional per-run cost ceiling in US dollars (loopConfig.maxCost), and maxTokens per step. Identity is a declared property of each resource, a tenancy strategy of internal, tenant-isolated or end-user-isolated evaluated before execution, so every cost figure arrives attached to a customer. The monthly spend cap is organization-level; a per-customer allowance comes from the metered figures and is enforced at your entry point.

Eval suites with judge scoring, human review and record-level comparison are what let a downgrade ship.