AI agent observability: every run traced, per tenant
What AI agent observability means, why agent runs are harder to trace than requests, what a useful trace contains, and what changes when customers use it.
AI agent observability is the ability to reconstruct what an agent did on a single run: which model calls it made and with what input, which tools it called with what arguments, what each tool returned, how long every step took and what it cost. Request logging records that a call happened; agent observability records the chain of decisions between the user's message and the final answer. For an agent your customers use, every one of those facts also has to carry the tenant and the end user it ran for.
This page covers why that is harder than it sounds, what a trace has to contain to be useful during an incident, and what changes once the agent is customer-facing and multi-tenant. Model-level metrics on their own (token counts, provider latency, cost per call with no run structure) are covered at LLM observability; where the agent should run at all is the subject of AI agent platform.
Why agent observability is hard
Five failure modes account for most of the pain.
The unit of work is a run, not a request
One user message can fan out into a dozen model calls and tool calls over several seconds or minutes, and an HTTP log gives you a dozen entries with no parent. Correlating them by timestamp works in development and fails in production the first time two tenants' runs overlap. A trace needs a root span per run and a parent for every child, or the run cannot be reassembled after the fact.
Failures return 200
A model that skipped the tool it was supposed to call and answered from its training data returns a well-formed completion with no error code. A tool that returned an empty array because the query was malformed also returned 200, and the model read the empty result as "nothing found" and said so to the user. Nothing in a status-code dashboard moves. The only evidence is in the arguments and results of individual calls, which is why finding silent failures is a trace-reading exercise rather than an alerting one.
Loops hide inside aggregates
An agent that calls the same search tool eleven times in one turn, because the tool keeps returning an empty result and the model keeps rephrasing, shows up on a latency chart as a worse p95. Per-run, it is obvious: the same span name eleven times, with arguments that differ by a word. The mechanism, why models retry on empty results and how to bound it, is worked through in agent infinite loops.
Tool arguments are the evidence, and they are usually dropped
Most instrumentation records that a tool was called and how long it took, and omits the arguments, because they can hold personal data or be large. The argument is the one fact that tells you the model formed the wrong query. Dropping it turns "the model searched for the customer's company name instead of their account ID" into "search tool, 340 ms, ok".
Cost lands on the wrong thing
Provider invoices arrive per API key. A single run spreads its cost across several calls under that key, with cached and uncached input tokens priced differently and reported in a different shape by each provider. Without per-call token accounting attached to the run, the question "what did tenant A's agent cost last week" has no answer better than a proportional guess.
How it works: the span tree
A useful agent trace is one tree per run. The root span represents the run and carries the facts that never change during it: the trigger (a chat surface, an API call, a schedule), the agent and its version, the tenant and the end user. Under it, a span per turn, and under each turn a span per model call and per tool call, in the order they happened. Flow-style agents add a span per step with the input variables it read and the output variable it wrote.
| Span level | What it records | What it answers |
|---|---|---|
| Run | Trigger, agent id and version, tenant id, end-user id, terminal status, total latency, total cost | Who this ran for, how it ended, what it cost |
| Turn | User input, assistant output, iteration count, model | What the user saw at each exchange |
| Model call | Full prompt sent (system and messages), response, stop reason, cached and uncached tokens, latency, cost | Why the model answered the way it did |
| Tool call | Tool name, arguments, result or error, duration, approval state if gated | What the model asked for and what it actually got |
| Step | Step type, input variables, output variable, branch or loop iteration | Where a deterministic pipeline diverged from expectation |
Two properties of this tree matter more than the rest. First, tool arguments and results are recorded in full, with a redaction policy applied at write time rather than by omitting the field. Second, the tenant and end-user identifiers are attributes on the root span, set by whatever executed the run, so that "show me every run for end user 8812 in tenant acme in the last hour" is an attribute filter and never a text search through metadata.
Cost and latency roll up from the leaves. Each model-call span carries its own token counts and price; the run total is a sum, and a per-tenant figure is a sum over root spans with that attribute. Keeping cached and uncached tokens separate at the leaf is what makes the number reconcile with the provider invoice.
Getting an agent built anywhere into the same view
The tree above does not depend on a framework. OpenTelemetry already models it: a trace is a tree of spans, spans carry attributes, and OTLP is the wire format every backend accepts. An agent written on LangChain, the Vercel AI SDK or a hand-rolled loop can export the same shape, and the semantic conventions for generative AI spans (the gen_ai.* attribute namespace) give model calls a common vocabulary for model name, token counts and finish reason. Those conventions are still at Development stability, so pin the version you targeted.
Exporting is configuration rather than code. A typical OTLP/HTTP setup is three environment variables:
export OTEL_EXPORTER_OTLP_ENDPOINT="https://collector.example.com/v1/otel"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_RESOURCE_ATTRIBUTES="service.name=support-agent,deployment.environment=production"
What the framework's instrumentation does not do for you is attribute the run. Set a root span per run, put the tenant and end-user identifiers on it, and make sure delegated or nested calls stay under that root instead of starting a new trace. The step-by-step for common frameworks is in instrumenting an external agent; the general model of spans, attributes and sampling is in agent tracing.
What changes when the agent is customer-facing and multi-tenant
Everything above holds for an agent your own team runs. Three things change when the agent sits inside your product and your customers use it.
Attribution has to be produced at execution time. A team-internal agent has one entry point and one place to set the tenant. A customer-facing agent is reached from a web widget, from Slack, from an API key a customer's backend holds, from a scheduled job and sometimes from another agent, and each is a place where the identifier can be forgotten. When the runtime that executes the run records the identity the request authenticated with, coverage is complete by construction; when calling code annotates it, coverage is as good as the least careful entry point.
Isolation becomes a property of the store, not a dashboard filter. A customer who asks "show me my usage" or "prove that another tenant never saw our prompts" is asking about the data. If the answer is "we filter by a metadata field in the UI", the answer is no. The identifier has to be set before the run starts, checked against the caller's actual scope, and used as the partition key for traces, logs and cost.
Cost per tenant turns from a report into a control. "What did tenant A cost last month" is a query. "Tenant A's agent is on its tenth call to the same tool in one turn, stop it before the turn finishes" is a runtime limit that a trace store cannot enforce because it sees the run after it ended. Per-turn tool-call budgets and per-tool call caps belong in the thing that runs the agent, and the trace is how you tune them.
Two smaller shifts follow. Traces now contain customer data, so PII redaction and logging verbosity need to be set per customer rather than per deployment, and applied before the span is written. And support changes shape: a customer reports "the assistant told me my order was cancelled", and someone has to find that run by end-user id, read the tool result that said status: "cancelled" for a different order, and turn it into a regression case. That loop is laid out in debugging an AI agent.
Where Runtype fits
Runtype gives a software company one place where every agent run is traced, priced and filed under the customer it ran for, whether or not the agent runs on Runtype.
Start by sending traces. Any OpenTelemetry-instrumented agent, a hand-written loop, LangGraph, the Vercel AI SDK, Flue or the Cloudflare Agents SDK, exports OTLP/HTTP to https://api.runtype.com/v1/otel, naming the target agent with an x-runtype-agent-id header or a runtype.agent.id resource attribute. That gets you the Runs view, the trace tree, structured logs, token usage and a display-only cost estimate. Your provider still bills you, an imported run is not a Runtype execution, and two instrumentations pointed at the endpoint double tokens and cost.
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.runtype.com/v1/otel"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${RUNTYPE_API_KEY},x-runtype-agent-id=${RUNTYPE_AGENT_ID}"
A run whose spans carry the GenAI content attributes has a readable transcript and can be captured as an eval case. Setup is at reporting external telemetry.
Three further steps stack on that, each optional. Register the agent as an external agent whose endpoint speaks Runtype's unified stream or A2A, and Runtype calls it: tool calls and cost are recorded per run, and the same agent can be tested from the dashboard, embedded in the open-source Persona chat widget, added to a product as a capability and put on schedules. Point it the other way instead, and an MCP surface exposes flows, agents, records and tools to the loop you already have. Rebuild one capability natively when a suite harvested from real runs can prove parity for it.
Identity is a declared property of each resource: a tenancy strategy of internal, tenant-isolated or end-user-isolated with an assurance floor of asserted or verified, evaluated before execution starts (end-user identity). Every trace and every cost figure is filed under the tenant and end user the request ran for, so per-tenant usage is a query over an attribute the runtime set. PII redaction and logging verbosity are policies set per product, surface or agent and resolved at dispatch.
For agents Runtype executes, the loops named above are bounded while they run: maxToolCalls per turn (default 10, maximum 100), maxTurns between 1 and 100, an optional per-run cost ceiling, and a 30-second default tool timeout. A tool that needs a human waits behind an approval gate, and the reason the approver reads is the agent's own claim, shown for context and never used as a control signal. The trace is what tells you where to set each of those numbers.
Frequently asked questions
- What is the difference between AI agent observability and LLM observability?
- LLM observability is about the model call: tokens in and out, latency, cost, and the prompt and completion text. AI agent observability is about the run that contains those calls: the order of model calls and tool calls, the arguments each tool received, what it returned, which branch the agent took, and how the run ended. You need both, but only the second one explains why an agent did something.
- What should an agent trace contain?
- One tree per run, with a span per turn, per model call and per tool call. Each model call carries the full prompt it was sent, the response, the stop reason, token counts split into cached and uncached, latency and cost. Each tool call carries the tool name, the arguments the model produced, the result or error, and its duration. The run span carries the trigger, the agent version, the tenant and the end user.
- Can I use OpenTelemetry to trace an AI agent?
- Yes. OpenTelemetry spans nest naturally into a run tree, and most agent frameworks either export OTLP directly or have an instrumentation package that does. The two things to get right are a single root span per run, so the tree is not split across exports, and tenant and end-user identifiers as attributes on that root span rather than in free-text metadata.
- How do I find out why an agent looped?
- Open the run trace and look at the tool-call spans in order. A loop almost always shows as the same tool called repeatedly with the same or nearly the same arguments, and the result of each call explains why the model retried: an empty array, an error string the model treated as a hint, or a result that did not contain the field the model expected. Aggregate dashboards hide this because a loop of ten calls looks like one slow request.
- Why is tenant attribution harder for agents than for ordinary services?
- An ordinary service has one request path and one place to set the tenant. An agent your customers use is reached from a web widget, Slack, an API key held by a customer backend and a scheduled job, and any one of those can forget to set the identifier. If calling code annotates it, coverage is only as good as the least careful entry point. If the runtime that executes the agent records it, coverage is complete.