How to instrument an agent you built yourself

Instrument the agent where it already runs, and export the spans somewhere else. That means one OpenTelemetry tracer created at startup, a span for each turn of the loop, a child span for each model call and each tool call, tenant and end-user identifiers attached to all of them, and an OTLP exporter pointed at whichever backend you want to read.

The sample below is Python with the vendor-neutral SDK. None of it is tied to a backend: the same span tree lands in Jaeger, Grafana Tempo, Langfuse, Phoenix or a commercial vendor, because the wire format is OTLP and the attribute names come from the OpenTelemetry GenAI semantic conventions.

The situation this is written for

The agent exists. It is a while loop around a chat completions call with a tool dispatch table, or a LangGraph app, or a Vercel AI SDK route handler streaming into a browser. Porting it onto a platform to get a trace view is a trade almost nobody accepts, and it is not required.

What usually exists for visibility is one JSON log line per request holding the final answer, plus a monthly provider bill. A customer reports that the agent gave a wrong answer at 14:12, and you have the answer without the eight steps that produced it: which tools ran, what they returned, how many times the model was called, which turn burned the token budget.

The pressure gets worse when the agent sits inside your product and many customers use it. Spend has to be attributable to a tenant, a failing turn has to be findable by the end user who reported it, and one customer's traces should not be the way you debug another's. Those are properties of the identifiers you attach at the entry point, so they are cheap to add now and expensive to backfill.

How to add observability to a LangChain or custom agent

1. Decide the span tree before writing code

Three levels cover almost every agent. One span for the run (the user's request, from first byte in to final answer out), one child per model call, one child per tool call. Nesting follows causality: a tool span is a child of the model call that requested it, or of the run span if your loop dispatches tools after the model returns.

Give each span a name of the form <operation> <target>, which is what the GenAI conventions ask for: chat gpt-4.1-mini, execute_tool search_orders, invoke_agent support-agent. Consistent names are what let a backend group nine executions of the same tool and show you that one of them is slow.

2. Create one tracer and one exporter at startup

Do this once per process, not per request. The identity processor below is the mechanism that stops every future call site from having to remember the tenant.

from opentelemetry import baggage, trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

IDENTITY_KEYS = ("tenant.id", "end_user.id")


class IdentitySpanProcessor(SpanProcessor):
    """Copy tenant and end-user identity from baggage onto every span."""

    def on_start(self, span, parent_context=None):
        for key in IDENTITY_KEYS:
            value = baggage.get_baggage(key, parent_context)
            if value is not None:
                span.set_attribute(key, value)


provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
provider.add_span_processor(IdentitySpanProcessor())
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("support-agent")

OTLPSpanExporter() with no arguments reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS from the environment and appends /v1/traces to the endpoint. Keeping the destination in configuration rather than code is what makes a second backend a deployment change.

3. Set identity once, at the entry point

Put the tenant and end user in baggage at the edge of the request, before the loop starts. The processor from step 2 then stamps them on every span the request creates, including spans from library instrumentation you did not write.

from opentelemetry import context


def handle_request(question, tenant_id, end_user_id):
    ctx = baggage.set_baggage("tenant.id", tenant_id)
    ctx = baggage.set_baggage("end_user.id", end_user_id, context=ctx)
    token = context.attach(ctx)
    try:
        with tracer.start_as_current_span(
            "invoke_agent support-agent",
            attributes={
                "gen_ai.operation.name": "invoke_agent",
                "gen_ai.agent.name": "support-agent",
            },
        ) as run:
            messages = [{"role": "user", "content": question}]
            for turn in range(MAX_TURNS):
                response = call_model(messages, turn)
                choice = response.choices[0]
                if not choice.message.tool_calls:
                    run.set_attribute("app.turns_used", turn + 1)
                    return choice.message.content
                messages.append(choice.message)
                for call in choice.message.tool_calls:
                    messages.append(run_tool(call))
            run.set_attribute("app.turn_limit_hit", True)
            raise TurnLimitExceeded(MAX_TURNS)
    finally:
        context.detach(token)

Baggage is a header, not a local variable, so the same values travel to downstream services when the default propagators are active (OTEL_PROPAGATORS defaults to tracecontext,baggage). Treat it as visible to anything you call, and keep secrets out of it.

4. Wrap each model call and record what the provider returned

The request attributes describe what you asked for. The response attributes describe what actually happened, which is the half that answers cost and truncation questions later.

def call_model(messages, turn):
    with tracer.start_as_current_span(
        "chat gpt-4.1-mini",
        kind=trace.SpanKind.CLIENT,
        attributes={
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": "openai",
            "gen_ai.request.model": "gpt-4.1-mini",
            "gen_ai.request.temperature": 0.2,
            "app.turn": turn,
        },
    ) as span:
        response = client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=messages,
            tools=TOOL_SCHEMAS,
            temperature=0.2,
        )
        usage = response.usage
        details = usage.prompt_tokens_details
        span.set_attributes(
            {
                "gen_ai.response.model": response.model,
                "gen_ai.response.id": response.id,
                "gen_ai.usage.input_tokens": usage.prompt_tokens,
                "gen_ai.usage.output_tokens": usage.completion_tokens,
                "gen_ai.response.finish_reasons": [c.finish_reason for c in response.choices],
                "gen_ai.usage.cache_read.input_tokens": details.cached_tokens if details else 0,
            }
        )
        return response

Two of those are easy to skip and painful to lack. gen_ai.response.model is the id the provider billed, which differs from the id you sent whenever an alias resolves to a dated snapshot. Cached input tokens are priced differently from fresh ones, so a cost figure computed from input_tokens alone overstates spend on any agent with a stable system prompt.

5. Wrap each tool call, including the failures

A tool that raises should end up as an error span and a message the model can read, rather than an exception that kills the turn. Returning the error keeps the loop alive and puts the failure in the trace where you can count it.

import json

from opentelemetry.trace import Status, StatusCode


def run_tool(call):
    with tracer.start_as_current_span(
        f"execute_tool {call.function.name}",
        attributes={
            "gen_ai.operation.name": "execute_tool",
            "gen_ai.tool.name": call.function.name,
            "gen_ai.tool.call.id": call.id,
        },
    ) as span:
        arguments = json.loads(call.function.arguments)
        span.set_attribute("app.tool.argument_keys", sorted(arguments))
        try:
            result = TOOLS[call.function.name](**arguments)
        except Exception as exc:
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            span.record_exception(exc)
            result = {"error": str(exc)}
        span.set_attribute("app.tool.result_bytes", len(json.dumps(result)))
        return {
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        }

app.tool.result_bytes and app.tool.argument_keys are custom attributes, and they earn their place. An empty result that the model reads as a failed call is the usual cause of a loop calling the same tool six times in one turn, and a size of two bytes in the trace names that cause immediately. The conventions do define gen_ai.tool.call.arguments and gen_ai.tool.call.result for the full values, both opt-in; these two record shape without carrying the payload.

6. Keep one logical call intact across retries

Provider SDKs retry on their own, so a 40-second model call can be three attempts and two backoffs hiding inside one span. Open a child span per attempt under the logical call, and put the attempt number on it. The parent then carries the latency the user felt, and the children explain it.

Cross-process work needs the opposite move. When a turn hands off to a queue, a cron worker or another service, the trace breaks unless you carry the context in the payload:

from opentelemetry.propagate import extract, inject

carrier = {}
inject(carrier)
queue.publish({"job": job, "otel": carrier})

def consume(message):  # worker side
    with tracer.start_as_current_span(
        "invoke_agent support-agent", context=extract(message["otel"])
    ):
        ...

7. Sample traces deliberately, and keep cost off that path

Head sampling is one environment variable and it drops whole traces uniformly, which is fine while volume is low:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
export OTEL_EXPORTER_OTLP_ENDPOINT="https://collector.example.com"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${OTEL_TOKEN}"

The problem with a ratio sampler is that it decides before it knows anything, so the failed turn you need is dropped with the same probability as a boring one. Run the OpenTelemetry Collector with the tail_sampling processor once volume forces the issue: it buffers a whole trace and then applies policies, so you can keep every trace with an error status or a duration above a threshold and sample the rest at a few percent. Cost and usage should not ride on either mechanism, for the reason in the FAQ below.

Attributes worth standardizing on

The GenAI semantic conventions cover most of what an agent produces. Every attribute in them still carries Development stability, the label OpenTelemetry uses for a name that can change, and several already have: gen_ai.system was replaced by gen_ai.provider.name, and gen_ai.prompt and gen_ai.completion were deprecated with no replacement. The conventions also moved out of the main semantic-conventions repository into a dedicated GenAI one in release v1.42.0 (June 2026), and that repository has no tagged release yet, so pin the instrumentation package version you install and read its changelog on upgrades.

Prompt and completion content is deliberately absent. What to store, where, and for how long is a separate decision covered in what to log for an LLM feature.

Two span layouts that waste the effort

The first is one span per run. You get a duration, a status and nothing else, so the trace tells you the request took 41 seconds without telling you that 38 of them were a retail-inventory tool timing out twice. If a span cannot be attributed to a single model call or a single tool call, it cannot answer the question you opened the trace to answer.

The second is one span per token or per stream chunk. A 900-token answer becomes 900 spans, the exporter queue backs up, the backend charges you for the volume, and the trace view is unreadable. Streaming deltas belong nowhere in the span tree; record time-to-first-token as an attribute on the model span and be done. The failure modes a good tree does catch are collected in debugging an AI agent and in the wider hub on AI agent observability.

Framework notes

LangChain and LangGraph emit their own run tree through the callback system rather than OpenTelemetry directly. Two community packages convert those callbacks into OTel spans: OpenInference's openinference-instrumentation-langchain and Traceloop's opentelemetry-instrumentation-langchain. The LangSmith SDK exports the same runs over OTLP when you set LANGSMITH_OTEL_ENABLED=true beside LANGSMITH_TRACING=true and point OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS at your backend; LANGSMITH_OTEL_ONLY=true sends them there and skips LangSmith itself, and the Python SDK documents 0.4.25 or newer for that path. Either way, open your own run span around .invoke() so the framework's spans nest underneath it and inherit your baggage.

The Vercel AI SDK carries its own telemetry, and version 7 rewired it. Span collection moved out of the ai package into @ai-sdk/otel, registered once at startup, after which every call emits spans by default. The per-call option lost its experimental_ prefix and is now telemetry, its metadata field is gone, and identity values ride runtimeContext and reach the exporter only for the keys named in telemetry.includeRuntimeContext, arriving as ai.settings.runtimeContext.* beside the standard gen_ai.* attributes.

import { generateText, registerTelemetry } from 'ai'
import { OpenTelemetry } from '@ai-sdk/otel'

registerTelemetry(new OpenTelemetry())

const result = await generateText({
  model: openai('gpt-4.1-mini'),
  messages,
  tools,
  runtimeContext: { tenantId, endUserId },
  telemetry: {
    functionId: 'support-agent-turn',
    includeRuntimeContext: { tenantId: true, endUserId: true },
  },
})

For a hand-rolled loop, the Python sample above is the whole implementation. A comparison of how much of this each framework gives you before you write any of it is at agent framework comparison, and the span-tree design itself goes deeper in agent tracing.

Where this gets easier

Runtype accepts OpenTelemetry traces at https://api.runtype.com/v1/otel (standard OTLP, /v1/traces) from an agent running anywhere, so the spans you just built become the input to trace review, cost reporting per execution and eval suites without moving the agent off your infrastructure. An external agent can also be registered over A2A or a streaming endpoint, which lets those traces be joined to eval runs against the same agent rather than sitting in a separate viewer. Setup is documented at reporting external telemetry.