Pydantic AI in production: evals, cost and tenancy

Pydantic AI gives a Python service three things that are painful to retrofit: dependencies with types, model output validated against a Pydantic model, and an agent definition that does not name a provider. What it leaves to you is everything around the run. Tenant context, cost per customer, eval sets that outlive a model upgrade, and approval on risky tools all live outside the library.

Nothing below argues for replacing it. The agent definition is the part worth keeping, and most of the work of turning a Pydantic AI service into a customer-facing feature happens on the other side of agent.run().

What Pydantic AI does well

The deps_type parameter on Agent fixes one type for everything a run needs: a database handle, an HTTP client, the caller's identity. Tools and dynamic system-prompt functions receive a RunContext[Deps] and read ctx.deps, so there is exactly one channel into the run and the type checker follows it end to end. Against a framework that hands tools an untyped state dict, this removes a whole class of "who put that key there" bugs.

Output is the other half. output_type takes a Pydantic model and the validated object is what the run returns, so the boundary between model and application is a schema rather than a string you parse. When validation fails, the framework feeds the error back to the model as a retry instead of raising, bounded by the agent's retries setting, which defaults to one and can be overridden per tool. A tool can ask for the same treatment by raising ModelRetry with an instruction the model can act on.

Model choice stays late. An agent is declared with a string like anthropic:claude-sonnet-4-0 or a model object, and the same agent runs against a different model by passing one argument at call time. Provider differences in tool calling and structured output are the library's problem rather than yours, which is what makes a model upgrade a config change instead of a rewrite.

Testing is unusually good for this category. TestModel and FunctionModel exercise the entire agent graph, tool calls included, without a provider request, and Agent.override() swaps the model and the deps inside a test. Setting pydantic_ai.models.ALLOW_MODEL_REQUESTS = False makes any request to a non-test model raise, so a test suite cannot quietly spend money on a forgotten fixture.

Instrumentation and evals are first-party rather than bolted on. Turning instrumentation on emits OpenTelemetry spans that follow the GenAI semantic conventions, and Logfire renders a run as a tree with prompts, tool calls and token usage. The pydantic_evals package ships beside it with Case and Dataset, built-in evaluators including LLMJudge and ToolCorrectness, and a report you can assert on in CI.

The seam

The library's contract ends at the boundary of one agent.run() call inside one Python process. That is a defensible place to stop, and it is why the library stays small and testable. A feature your customers use has obligations on the far side of that boundary, and they do not go away because the agent is well typed.

• Identity is a value, not a policy. deps carries a tenant id because your code put it there. Nothing rejects a run whose deps were assembled from a stale session, a background job that defaulted the field, or an internal script running with a shared key.
• Cost stops at one run. RunUsage carries token counts and a best-effort cost in USD, priced by Pydantic's own genai-prices package, which returns None for a model whose pricing it does not carry. Turning per-run estimates into dollars per tenant per month is still yours: a write path to somewhere durable, and a decision about what to do when one customer's spend triples overnight.
• Evals run where you run them. pydantic_evals is a library, so the dataset is a file in your repo and the run happens in CI or on a laptop. There is no default route from a production run that went wrong to a new case in that dataset, which is the route that actually keeps an eval set honest.
• Surfaces multiply the entry points. One agent, but web chat, Slack, an API key a customer's own backend calls, and a nightly job are four applications you write, and every one of them is a place to forget the tenant.

Approval is where the boundary bites hardest. A tool that charges a card or emails a customer wants a human decision in the middle of a run, which means the run has to survive minutes or hours of waiting. Pydantic AI ships durable-execution capabilities for Temporal, DBOS, Prefect and Restate, attached to an agent through its capabilities argument, so the loop can sit on a workflow engine instead of an asyncio task. That answers the durability question by handing you a second system to operate.

How to close it

Build deps in one place, from verified identity

The dependency object is the only channel into a run, so make it the only place identity is decided. Construct it in a single factory that takes a verified token or session and returns the typed object, then have every tool filter by what it finds there.

from dataclasses import dataclass
from pydantic_ai import Agent, RunContext

@dataclass
class Deps:
    tenant_id: str
    end_user_id: str
    db: Database

agent = Agent(
    "anthropic:claude-sonnet-4-0",
    deps_type=Deps,
    output_type=TicketTriage,
)

@agent.tool
async def search_tickets(ctx: RunContext[Deps], query: str) -> list[Ticket]:
    # tenant_id comes from deps, never from the model
    return await ctx.deps.db.search(tenant_id=ctx.deps.tenant_id, query=query)

@agent.system_prompt
def workspace_rules(ctx: RunContext[Deps]) -> str:
    return f"You are answering inside workspace {ctx.deps.tenant_id}."

One rule carries most of the weight: no tool signature takes a tenant id as a parameter. Anything in a tool's parameter schema is a value the model supplies, and a model that can supply a tenant id can supply the wrong one. Keeping it in deps means the worst a confused model can do is search its own tenant badly.

Export the spans somewhere that outlives the process

Instrumentation already produces OpenTelemetry spans, so the work is choosing a destination and attaching the attributes you will want to group by later. The standard environment variables are enough for most collectors, and the same spans can go to more than one backend while you decide.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://collector.example.com/v1/otel"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${OTLP_TOKEN}"
export OTEL_SERVICE_NAME="ticket-triage"

import logfire

logfire.configure(send_to_logfire=False)
logfire.instrument_pydantic_ai()

The Logfire SDK is one of two documented routes. Without it, build a plain OpenTelemetry TracerProvider with an OTLP exporter, call set_tracer_provider, then Agent.instrument_all(), and the same GenAI spans reach the same collector.

Set tenant_id and end_user_id as span attributes at the top of the request, before the agent runs, so every child span inherits the grouping key. Decide the prompt and tool-argument capture setting deliberately: those payloads are where customer data ends up, and a redaction policy is cheaper to add now than after a year of retained spans. A walkthrough of pointing an externally-run agent's spans at a platform that will read them is at instrumenting an external agent.

Decide what stays in the framework

Most retrofits go wrong by moving too much. The agent code is the part with the best tests and the clearest types, so leave it alone and put the operational concerns beside it.

If you are still choosing, the tradeoffs against other Python and TypeScript options are in agent framework comparison, and the graph-shaped alternative is covered in LangGraph alternatives. The category view of what the surrounding layer contains is at AI agent platform.

Where Runtype fits

Runtype is where a Pydantic AI service's runs, cost, evals and tenant rules can live while the agent definition stays as written. The first way in is sending traces. Instrumentation already emits OpenTelemetry spans that follow the GenAI semantic conventions, so pointing the standard OTLP variables at https://api.runtype.com/v1/otel turns them into runs with a trace tree, structured logs, tool calls with arguments and results, token usage and a display-only cost estimate. Your provider still bills you, an imported run is a record rather than a Runtype execution, and exactly one instrumentation should point at the endpoint, because two doubles the tokens and the cost.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.runtype.com/v1/otel"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${RUNTYPE_API_KEY},x-runtype-agent-id=${RUNTYPE_AGENT_ID}"

Setup is at reporting external telemetry. Because those spans carry the GenAI content attributes, a run has a transcript, and a run that went wrong can be captured as an eval case rather than invented. That is the route pydantic_evals leaves open: a case promoted from a recorded execution, judged by a model with human review of individual scores.

The second way in registers the agent so Runtype can call it. An external agent points at an endpoint of yours that speaks Runtype's unified stream or A2A, and the Python service behind it is unchanged. It can then be embedded in the open-source Persona chat widget, added to a product as a capability, reached through web chat, Slack, a REST API, SMS, iMessage, MCP and A2A, and put on a schedule, with tool calls and cost recorded per run. An A2A endpoint can also serve as an eval suite target, where cases that replay recorded tool activity are skipped.

The third way points in the other direction: an MCP surface exposes a product's flows, agents, records, skills and tools as MCP tools, so the Pydantic AI agent stays the orchestrator and reaches Runtype for the parts worth centralizing, with deps_type and output_type untouched. The fourth moves one capability into a native flow or hosted agent, taken once the suite harvested from real runs can prove parity, and gated by runtype eval run in CI, which returns a non-zero exit code on a regression. A full rebuild stays available and is never the price of the first three.

The identity contract is the part deps cannot cover on its own. Each resource declares a tenancy strategy of internal, tenant-isolated or end-user-isolated with an assurance floor of asserted or verified, evaluated before execution, so a request whose identity scope falls short is rejected before it runs. Every trace and every cost figure is filed under the tenant and end user the run served, and long-term memory, when enabled, is keyed per agent, tenant or end user (end-user identity). Approval gates, per-turn maxToolCalls and loopConfig.maxTurns sit on the same side of the boundary, and the runtime runs as managed cloud or self-hosted on your own infrastructure.