Evals and cost attribution for an OpenAI Agents SDK app

Keep the OpenAI Agents SDK. Its five primitives (agents, handoffs, guardrails, sessions and tracing) describe a working agent in a few hundred lines of Python or TypeScript, and the abstraction surface stays small enough to read in an afternoon. A customer-facing deployment adds four things around it: tenant identity enforced at the tool, cost attributed to an account, approval gates the SDK does not model, and eval sets built from real traffic.

This page is for a team that already chose the SDK, has it working, and now has paying accounts behind it. Nothing below asks you to move the loop or to trade a small abstraction surface for a larger framework.

What the OpenAI Agents SDK does well

The package is deliberately small. An Agent is a name, a set of instructions, a model, a list of tools and a list of handoffs, and Runner.run drives the loop until the agent produces a final output or a guardrail halts it. Tools are ordinary functions wrapped with function_tool, which reads the signature and docstring to build the schema the model sees. Little is hidden, and a stack trace usually lands in your own code rather than three layers of framework.

Handoffs are the idea most worth keeping. A handoff is exposed to the model as a tool call, so routing from a triage agent to a billing specialist is a decision made inside the turn rather than a graph drawn in advance. The handoff() helper adds an on_handoff callback that fires when the transfer happens and an input_filter that trims which history the receiving agent sees. For a support agent that fans out into several specialists, that is a shorter description of the problem than nodes and edges.

Guardrails run beside the agent instead of inside the prompt. An @input_guardrail screens the incoming message before the expensive model runs, an @output_guardrail checks the final result, and either can set a tripwire that stops the run by raising InputGuardrailTripwireTriggered or OutputGuardrailTripwireTriggered rather than by returning a value your caller might ignore. Putting a cheap classifier in front of an expensive model is a well-worn shape, and the SDK makes it three lines.

Sessions remove the most tedious part of a chat loop. Pass one to a run and prior turns are prepended and new ones appended, so history stops being a list threaded through your own handlers. A local SQLiteSession ships in the box, alongside RedisSession, SQLAlchemySession, MongoDBSession, OpenAIConversationsSession for server-managed history, and an EncryptedSession wrapper.

Tracing is on by default, which is unusual and useful. Each run produces a trace with spans for model generations, tool calls, handoffs and guardrail checks, and related runs group under one identifier with metadata attached. The processors that receive those spans are pluggable, so a second destination is a registration call rather than a fork.

Model choice stays open. Other providers are reachable through a Chat Completions-compatible endpoint or through LiteLLM, installed as the openai-agents[litellm] extra, so a small model on the triage agent and a large one on the specialist is a per-agent setting. The Python and TypeScript packages track the same concepts, which matters when the service is Python and the product is Node.

The seam

The SDK ends at the process boundary. It describes an agent and runs it; everything about who the run belonged to, what it cost that account, and whether last week's prompt change helped sits above that line. Five gaps show up in roughly the same order for every team that puts one in front of customers.

Identity is a convention, not a boundary. A run takes a context argument, a local object of your own type that reaches every tool through a wrapper and is never serialized into the model's messages. That is exactly the right carrier for a tenant id, and it is also the only one: nothing verifies that a tool consulted it. A tool that filters by an id taken from the model's arguments rather than from the context compiles, passes every test written against a single tenant, and reads another account's rows the first week it ships.

Traces stop at the project that owns the API key. Span data lands in OpenAI's Traces dashboard under the organization and project of the exporting key, and an organization on a Zero Data Retention agreement cannot use the built-in tracing at all. Filtering by an attribute you set is a dashboard feature, so showing one customer their own runs, joining a trace against your application logs, and keeping a year of evidence for an audit each need the spans somewhere you control.

Usage is per run; cost per account is arithmetic you own. A finished run exposes token counts on result.context_wrapper.usage, with cached input tokens broken out under input_tokens_details.cached_tokens, and a price table turns those into money. Cached tokens are billed differently, so a single multiplication overstates a long conversation. No budget exists inside the loop, which means one account can spend a month of margin in one turn and you find out from the invoice.

Approval pauses the run, and the policy around it is yours. Both packages model the pause the same way: a tool declares needs_approval, the run stops with the pending calls in result.interruptions, and you approve or reject each one against a run state that serializes to a string and resumes on another machine. Hosted MCP tools use require_approval inside the tool config. What no version supplies is the policy: which accounts need review for which tools, a timeout when nobody answers, an audit record of who approved, and somewhere outside a terminal for a reviewer to see the request.

There is no eval loop. No dataset, no scorer, no experiment, no regression suite, and that omission is consistent with the SDK's scope. What fills the space is usually a folder of example prompts read by eye, which holds until a model version moves under a stable alias and nobody can say what changed.

How to close it

The work divides along a clean line. Agent definitions, handoffs, guardrails and the loop stay in the SDK. Identity, measurement, budget and review go in a thin layer around it, and none of that layer requires touching an Agent definition.

1. Put the tenant in the context object, then check it in the tool

Resolve the account at your HTTP edge, before any agent runs, and pass it down as a typed context object. Because that object never reaches the model, a tenant id inside it cannot be echoed, guessed or argued with.

from dataclasses import dataclass

from agents import Agent, RunContextWrapper, Runner, function_tool


@dataclass(frozen=True)
class TenantContext:
    tenant_id: str
    end_user_id: str
    entitlements: frozenset[str]


@function_tool
async def list_invoices(ctx: RunContextWrapper[TenantContext], status: str) -> list[dict]:
    """List invoices for the current account."""
    if "billing" not in ctx.context.entitlements:
        return []
    return await invoices_for(tenant_id=ctx.context.tenant_id, status=status)


result = await Runner.run(
    support_agent,
    "Where is my last invoice?",
    context=TenantContext(
        tenant_id="t_4471",
        end_user_id="u_88",
        entitlements=frozenset({"billing"}),
    ),
)

Two rules make this hold under review. No tool signature accepts a tenant, an account or a customer identifier as a parameter the model fills in, so the model has no way to name a tenant even if a prompt injection asks it to. And the tool pool is computed from entitlements before the run starts, since a tool the model can see is a tool it will eventually try. The wider list is in the tenant isolation checklist.

2. Register a trace processor and export the same spans over OTLP

Keep the built-in tracing. Add a processor with add_trace_processor so every span also goes to a destination you own (set_trace_processors replaces OpenAI's exporter instead of adding to it), with the tenant and end-user identifiers as attributes on the root span, where child spans inherit them.

Most teams do this with an OpenTelemetry exporter, since the span vocabulary is already close and the receiving side is then a normal collector:

OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com/v1/otel
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=support-agent

What goes on a span matters more than where it lands. Record the account, the end user, the surface the request arrived on, the agent name and version, the model id, and for every tool call its arguments and result. A trace missing the arguments tells you a tool ran and nothing about why it ran wrong.

3. Attribute cost at the run, then cap it inside the turn

Read the token counts off each finished run, price them with a table you keep beside the model configuration, and write the figure against the account before the request returns. Keep cached and uncached input tokens as separate columns, because the ratio between them is the first thing that moves when a prompt is reorganized.

A recorded number stops a bad month. Stopping a bad turn needs a counter checked as the loop runs: a cap on tool calls per turn, a cap on turns per run, and a spend ceiling that ends the run when it trips. That code belongs beside the account resolver rather than inside any agent.

4. Build the eval set from production traffic, not from imagination

Once traces land somewhere queryable, the runs themselves are the dataset. Pick the ones that went wrong, the ones a customer complained about, and a sample of ordinary successful ones, then freeze each as a case with its input, its context and the behavior you wanted. A case from a real conversation carries the messy phrasing and half-specified request a handwritten example never has. The method, including how to sample without filling the set with near-duplicates, is in building an eval set from production.

Score them on what you care about: whether the right specialist got the handoff, whether the tool was called with the arguments the request implied, whether a guardrail fired when it should have. Most have no single correct string, which is why a judge model with human review of the scores beats an exact-match assertion. The category is covered on the AI agent evals page.

5. Keep in the SDK

Leave agent definitions, handoffs, guardrails, sessions and the loop where they are. They are small, readable and maintained by the people shipping the models, and reimplementing them buys nothing. Move only the concerns that are about your customers: who is asking, what they may reach, what they cost, when a person has to approve, and how you know a change was an improvement. The same division applies to every framework in this category, which is the argument made at length in the agent framework comparison.

Where Runtype fits

Runtype is the serving and measurement layer around an agent your own process runs. Four ways in stack, and a rebuild stays optional.

Register the agent first: an external agent points at an endpoint speaking Runtype's unified stream (runtype-stream) or A2A, and Runtype calls your Runner loop from there. It then embeds in the open-source Persona widget, joins a product as a capability, answers on web chat, Slack, REST, SMS, MCP and A2A, and runs on a schedule, with cost recorded per run. An A2A endpoint can also be an eval suite target, and the SDK code is unchanged (bring your own agent).

Send traces next. One OTLP exporter pointed at https://api.runtype.com/v1/otel gives you the Runs view and trace tree, token usage and a cost estimate for display, since an imported run is not a Runtype execution and OpenAI still bills you. Spans carrying the GenAI content attributes add a transcript, capturable as an eval case. Point exactly one instrumentation at it; two double the numbers.

Serve it tools. An MCP surface publishes a product's flows, agents, records, skills and tools, so the Runner loop stays the orchestrator and calls Runtype for what is worth centralizing. Rebuild last: port the capability that keeps breaking into a flow once a suite harvested from real runs proves parity, then gate it with runtype eval run in CI, which exits non-zero on regression.

The layers exist for your customers. A resource declares a tenancy strategy of internal, tenant-isolated or end-user-isolated with an assurance floor of asserted or verified, evaluated before execution, and every trace and cost figure files under the tenant and end user it ran for.

An approval gate is the general pause the SDK leaves you to write: it holds a tool call for a human on a five-minute default timeout and resumes through the same dispatch, and maxToolCalls bounds a turn at 10 by default and 100 at most. The runtime can be self-hosted where the data cannot leave.