How to trace an agent run across models, tools and retries

Model the run as one span tree rather than a log stream. Open a single root span when the run starts, nest a span per turn, per model call and per tool call under it, and attach the tenant and end-user identity to the root so every descendant inherits it. Retries and streaming break the naive version of this, and each has a specific fix.

One user message, fourteen calls, no parent

A support agent answering "when did my last order ship?" might do fourteen things: two retrieval queries, four model calls across three turns, a tool call that times out and is retried, a subagent that condenses the thread, and a final generation. An APM waterfall shows fourteen HTTP spans, most of them POSTs to one provider hostname, sorted by start time and unrelated to each other.

Sorting by timestamp is enough while you are the only person using the agent. It stops working the first afternoon two tenants have overlapping runs, and it never worked for anything that continued after the HTTP response closed: a background summarisation, a queued follow-up, a webhook that resumed a paused turn.

The missing piece is parentage. Reading a trace during an incident means asking "which model call produced these arguments" and "what did the tool return before the model said that", and both questions are about edges in a tree. What a finished trace has to contain, and why agent runs are harder to reconstruct than requests, is covered at AI agent observability.

How do I trace a multi-step AI agent run

Emit one trace per run, with a root span that lives for the whole run and a child span for each unit of work inside it. The nesting is not the HTTP call graph. The parent of a tool span is the turn that decided to call it, even though the HTTP request to that tool was issued by a different layer of your code, and the parent of a subagent's spans is the tool span that delegated to it.

Build the trace in seven steps

1. Open exactly one root span, at the point the run is admitted

The root span belongs to whatever accepts the work: the chat surface handler, the API route, the queue consumer, the scheduler. It ends when the run reaches a terminal state, not when the HTTP response is flushed.

Two bugs account for most broken trees. One is a trace per turn, which turns a five-turn conversation into five unrelated traces and makes "what happened in this conversation" unanswerable. The other is losing context across an async boundary: work handed to a background task starts a fresh trace because the span context was not captured and re-activated in the worker. Both look fine in development, where runs are single-turn and synchronous.

2. Nest turn, model call and tool call under the root

Four levels cover almost every agent: run, turn, model call, tool call. Deterministic pipelines add a step level between turn and model call, and delegation adds a subagent span whose children are that child's own model and tool calls.

agent.run  8.4s  tenant.id=acme  user.id=8812  agent.version=17
├─ turn 1  4.1s
│  ├─ chat claude-sonnet  1.2s  in=2431 out=88   finish=tool_calls
│  ├─ tool search_orders  0.34s  args={"query":"Northwind Trading"}  result={"orders":[]}
│  ├─ chat claude-sonnet  0.9s  in=2624 out=112  finish=tool_calls
│  ├─ tool search_orders  attempt=1  30.0s  status=error  error.type=timeout
│  ├─ tool search_orders  attempt=2  0.41s  result={"orders":[{...}]}
│  └─ chat claude-sonnet  1.2s  in=3908 out=204  finish=stop
└─ turn 2  4.3s
   ├─ chat claude-sonnet  0.8s  in=4102 out=61   finish=tool_calls
   └─ tool summarise_thread  3.4s   (subagent)
      ├─ chat claude-haiku  0.6s  in=3711 out=140
      └─ tool fetch_thread  0.5s  args={"thread_id":"th_91"}

That shape answers questions a flat list cannot. The first search_orders returned an empty array in 340 ms with a success status, and the model kept going as if the account had no orders, which is the failure mode walked through in debugging an AI agent. The retry is visible as two siblings rather than one call that took 30 seconds.

3. Put identity on the root span and propagate the context

The facts that do not change during a run belong on the root: tenant id, end-user id, agent id and version, the trigger that started it, and a conversation id if there is one. Descendants inherit them at query time through the trace id, so repeating them on every child wastes storage without adding an answer.

Propagation across a process boundary is the part that gets dropped. A subagent running as a separate service, a tool implemented as an internal HTTP endpoint, or a queue message that resumes a paused turn all need the W3C traceparent header carried through, or their spans start a new trace and the subtree detaches silently.

4. Name attributes so a filter still works after a framework change

Attribute names are a schema. Pick them once, write them down, and apply them in every instrumentation you own, because the cost of getting this wrong is a trace store where the same fact lives under four keys.

OpenTelemetry publishes semantic conventions for generative AI spans under the gen_ai.* namespace. The model-call level carries gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model and gen_ai.response.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons; agent and tool spans add gen_ai.agent.name, gen_ai.tool.name and gen_ai.tool.call.id. Follow them where they exist, so that a backend which understands the vocabulary renders your spans without custom mapping.

Treat the specific names as version-pinned rather than stable, and record which version of the conventions your instrumentation targets. As of September 2026 every GenAI attribute is marked Development, OpenTelemetry has moved the set out of the main semantic-conventions repository into a dedicated one (release v1.42.0, June 2026), and names have already changed: gen_ai.system is deprecated and replaced by gen_ai.provider.name, while gen_ai.prompt and gen_ai.completion are deprecated with no replacement. Prompt and response content now rides two opt-in structured attributes, gen_ai.input.messages and gen_ai.output.messages, with instructions supplied outside the chat history in gen_ai.system_instructions.

Tenant and end user have no generative AI convention, because they are not a model concept. Use a namespaced application attribute you control, such as app.tenant.id, for the tenant. For the person the registry still offers two names: user.id in the newer user namespace, and enduser.id, which was deprecated in v1.28.0 and then reinstated in v1.31.0 alongside enduser.pseudo.id for a pseudonymous identifier. Both sit at Development stability, so pick one, write it down, and keep it out of free-text metadata blobs so it stays filterable.

5. Record each retry as a sibling span with an attempt number

A retry that is folded into its parent span shows up as one call that was slow. The first attempt's error, the reason it failed and the delay before the second attempt all disappear, and the aggregate view reports a healthy success rate over calls that succeeded on the third try.

Provider SDKs make this harder than it sounds, because most of them retry internally on 429 and 5xx responses before your code sees anything. You have two options: set the client's max retries to zero and handle retries in the layer that owns the span, or use whatever request hook the SDK exposes to emit a span per HTTP attempt. Whichever you pick, the attempt number and the error type belong on the span, so a run with three quiet retries is distinguishable from a clean one.

6. End a streaming span on the final chunk, not the first byte

Naive instrumentation ends the model-call span when the response object is returned. For a streamed response that happens at the headers, so an eight-second generation records as 200 ms, and the token usage is missing because providers send it in the last chunk.

Record two timings instead: time to first token, as an attribute or an event on the span, and total duration when the stream completes. Also decide what an abandoned stream looks like. A client that disconnects halfway leaves a span that will never see a final chunk, so set a status of cancelled with the partial output length rather than leaving it open until a timeout closes it with no data.

7. Sample whole runs, never individual spans

Sampling applied per span produces trees with holes, and a tree with holes is worse than nothing: the model call is there but the tool result it reacted to is gone. Make the decision once at the root, where the sampled flag propagates to every child, or defer it to a collector doing tail sampling over the complete trace.

Keep everything for the runs that matter regardless of rate: any run that errored, hit a turn or tool-call limit, waited on a human approval, exceeded a cost threshold, or belongs to a tenant currently under investigation. Then sample successful short runs hard. The volume in an agent product is dominated by the boring ones.

Make the trace findable from a customer complaint

A trace nobody can find during an incident is not observability. Store the trace id on the conversation record and on the message the agent produced, so a support ticket that names a conversation resolves to a trace in one query rather than a timestamp search across tenants.

The reverse direction matters as much. Filtering by end-user id and a time window is how most investigations start, which only works if that attribute was set on the root span by the code that admitted the request, not by a call that a background job or a second surface can forget. Which fields to capture at the model-call level, and how to redact them safely, is covered in what to log for LLM calls; wiring this up inside LangChain, the Vercel AI SDK or a hand-written loop is in instrumenting an external agent.

Where this gets easier

Every step above is instrumentation you write and then have to remember to keep writing, and the gaps show up as an empty span exactly when you need it. Runtype produces this structure for agents running on the platform, because the runtime executed each step: per-step input and output, each tool call with its arguments and result, latency per step, and cost per execution with cached and uncached tokens counted separately, all carrying the tenant and end-user identity the request authenticated with. For agents built elsewhere, it accepts standard OTLP traces at https://api.runtype.com/v1/otel, so a LangChain or Vercel AI SDK agent lands in the same view as one that runs natively (reporting external telemetry).