LLM observability across models, tools and retries

LLM observability is the ability to reconstruct any single model call after the fact: the exact prompt sent, the completion that came back, the stop reason, the token counts split into cached and uncached, how long it took to start and to finish, whether it was a retry, and what it cost. Aggregate dashboards of request rate and error rate describe the service; LLM observability describes the call. When the calls run inside a product your customers use, every one of those facts also has to say which tenant and which end user it ran for.

This page stays at the level of the model call: prompts, tokens, latency, retries and the tool calls a model asks for. The layer above it, where those calls are grouped into a run with turns and branches, is covered at AI agent observability, and the question of where the whole thing should execute is the subject of AI agent platform.

Why LLM observability is hard

Request logging assumes a status code means something and a body is small. Model calls break both assumptions.

The wrong answer arrives as a 200

A model that was asked to extract an order number and returned "I could not find an order number in the text" produced a valid completion, a stop finish reason and a normal latency. Your error rate did not move. The same is true of a completion cut off at max_tokens halfway through a JSON object, which a parser downstream turns into a fallback value rather than an exception. The evidence lives in the response body and the stop reason of individual calls, which is why finding silent failures in AI features is a matter of reading records rather than watching a graph.

Retries are invisible and billed in full

Provider client libraries retry 408, 429 and 5xx responses with exponential backoff, twice by default in the current Anthropic and OpenAI SDKs. Each retry resends the whole prompt and is billed if the provider accepted it before failing, but the application sees one call that took nine seconds and writes one line in its log. Without the attempt number and the failed responses recorded as their own entries, a week of intermittent overload shows up as a cost line nobody can explain and a p95 nobody can reproduce.

Token counts come back in a different shape from every provider

Cached input tokens are the single largest lever on cost, and no two providers report them the same way. Anthropic returns cache_read_input_tokens and cache_creation_input_tokens beside input_tokens; OpenAI nests cached_tokens under prompt_tokens_details in Chat Completions and input_tokens_details in Responses; Gemini reports cachedContentTokenCount. An integration that reads only the top-level input count misprices every call, and the monthly invoice reconciliation never closes.

One latency number hides three different problems

The time a model call takes is queue time at the provider, time to first token, and generation time proportional to output length. A slow p95 can be any of the three, and each has a different fix: fail over to another provider, shorten the prompt, cap the output. A single duration_ms field cannot tell them apart. Recording time to first token separately from total time, with the output token count beside both, turns the slow-call question into arithmetic.

The prompt you logged is not the prompt you sent

Most applications log the template variables, or the user message alone, and reconstruct the prompt from the template in source control when needed. The template changed the following Tuesday. The system prompt gained a paragraph, a retrieval step started injecting eight documents instead of five, and the record from Monday can no longer be replayed. The rendered prompt, in full, is the field that survives a template change; the rest of the list is in what to capture in LLM logs.

How it works: one record per model call

A useful LLM observability record is written when the call returns, by whatever made the call, and contains enough to replay the call and to price it.

In JSON, one record looks like this:

{
  "id": "call_01J9X4K2",
  "parentId": "step_extract_order",
  "tenantId": "acme",
  "endUserId": "8812",
  "provider": "anthropic",
  "model": { "requested": "claude-sonnet-latest", "served": "claude-sonnet-4-5" },
  "prompt": {
    "hash": "sha256:9f2c1a",
    "system": "You extract order numbers from support messages.",
    "messages": [{ "role": "user", "content": "Where is order AC-44810?" }]
  },
  "response": {
    "stopReason": "tool_use",
    "toolCalls": [{ "name": "lookup_order", "arguments": { "orderId": "AC-44810" } }]
  },
  "tokens": { "input": 1412, "cachedInput": 1180, "output": 61 },
  "timingMs": { "firstToken": 640, "total": 1210 },
  "attempt": 2,
  "retryOf": "call_01J9X4JZ",
  "costUsd": 0.0031
}

Three details in this record do most of the work. The retryOf link makes the failed first attempt findable and its cost countable. A stopReason of tool_use means the response is a request rather than an answer, and the tool call that follows should be recorded as a child of this call, so that a wrong argument is attributable to the prompt that produced it. And the cachedInput count lets the cost figure reconcile with an invoice that prices cached tokens at a fraction of the uncached rate.

Tool calls belong under the model call that requested them

A tool call is a model deciding to do something, and the argument it produced is the only evidence of what it decided. Record the tool name, the arguments exactly as the model emitted them, the result or error, and the duration, attached to the model call whose response requested it. The sequence (request, result, next call) is what you read when the model searched for the customer's company name instead of their account id. Nested into a run with one root per user interaction and a parent for every child, these records become a trace; the mechanics of spans, attributes and sampling for that shape are in agent tracing.

From records to alerts

Alerts on model calls work on ratios rather than counts. The share of calls ending in max_tokens rather than stop, the share with zero cached input tokens on a prompt that should be cached, the share that are attempt two or higher, and the share whose served model differs from the requested one are each a number that stays near zero until something breaks. A raw error rate catches provider outages and none of those.

What changes when the model calls are customer-facing and multi-tenant

The record described so far belongs to model calls a single team makes and reads. Three things change when the calls happen inside a product your customers use.

The tenant has to be on the record before the call is made. An internal tool has one entry point. A customer-facing feature is reached from a chat widget, from Slack, from an API key a customer's backend holds and from a scheduled job, and every one of those is a place the identifier can be forgotten. If the layer that makes the model call also holds the authenticated identity of the request, the record carries it by construction; if each caller annotates it, the least careful caller sets the coverage.

Prompts now contain customer data. A record with the full rendered prompt is exactly what you need to debug and exactly what a data-protection review will ask about, and the answer cannot be "we drop the prompt", because then the record is useless. Redaction policy and log verbosity have to be decided per customer, applied before the record is written, and applied to tool arguments and results as well as to the prompt, since an order lookup result is as personal as the message that triggered it. The narrower problem of stripping personal data from logs is covered at PII redaction in LLM logs.

Cost per tenant becomes a limit as well as a report. A per-tenant cost query needs the tenant on every record and the cached and uncached tokens split, which the per-call record provides. A per-tenant cap needs something that sees the call before it goes out, which a store of records cannot do. The cap belongs in the thing that makes the call; the records are how you choose the number.

One smaller shift follows. A customer asking why the assistant said their order was cancelled needs their records found by end-user id within minutes, so tenant and end user have to be indexed fields rather than keys inside a metadata blob, and the records found that way are the raw material for a regression case.

Where Runtype fits

Runtype keeps a record of this shape for the model calls it makes, and accepts calls made elsewhere.

Start by sending traces. Any OpenTelemetry-instrumented caller, a hand-written loop, LangGraph, the Vercel AI SDK, Flue or the Cloudflare Agents SDK, exports OTLP/HTTP to https://api.runtype.com/v1/otel, naming the target agent with an x-runtype-agent-id header, and gets the Runs view, the trace tree, structured logs and token usage (reporting external telemetry). The cost figure on an imported run is a display-only estimate computed from those token counts: your provider still bills you, and the imported run is not a Runtype execution. Pointing two instrumentations at the endpoint doubles both tokens and cost.

Three further steps stack on that, each optional. Register the agent as an external agent whose endpoint speaks Runtype's unified stream or A2A, and Runtype calls it: tool calls and cost are recorded per run, and the agent can be tested from the dashboard, embedded in the Persona chat widget, added to a product as a capability and put on schedules. Or point it the other way: an MCP surface exposes flows, agents, records and tools to the loop you already have. Rebuild one capability natively when a suite harvested from real runs can prove parity for it.

Identity is a declared property of each resource: a tenancy strategy of internal, tenant-isolated or end-user-isolated with an assurance floor of asserted or verified, evaluated before any model call is made (end-user identity). Every record and every cost figure is filed under the tenant and end user the request ran for. PII redaction and logging verbosity are policies set per product, surface or agent and resolved at dispatch.

For calls Runtype makes, the runtime writes the record: per-step input and output, latency per step, every tool call with the arguments the model produced, and cost per execution and per batch with cached and uncached tokens kept separate. The same runtime holds the limits that turn a cost report into a control: maxToolCalls per turn (default 10, maximum 100), maxTurns between 1 and 100, an optional per-run cost ceiling, and a 30-second default tool timeout. Per-step model selection means the requested and the served model are both known to the runtime that wrote the record.