AI agent evals: prove it works before you ship it

An AI agent eval is a repeatable test that runs the agent against a fixed input and grades what happened: the final answer, the tools it called, whether it stayed inside the data it was given, and whether it followed the rules the business set. LLM evaluation is the practice of building those tests, keeping them honest as the agent changes, and using their results to decide whether a change ships. For an agent your customers talk to, the suite is the release gate, because nobody on your team reads most of its conversations.

Why evaluating an AI agent is hard

Five properties of an agent each break a specific kind of test that ordinary software relies on.

The same input produces different outputs

Sampling above temperature zero varies the wording on every run, and even at zero the provider's batching and floating-point behavior can change a token. A test that asserts exact string equality fails on a Tuesday for no reason, so teams loosen it until it asserts nothing. The trajectory varies too: a search tool that returns a different top result changes every step after it. The mechanics and the workarounds are in why LLM output is non-deterministic and how to test around it.

There is often no single right answer

A support reply, a summary, a draft email: two good answers to the same input differ in structure and wording, and a grader that measures similarity to one reference answer punishes the better answer for phrasing it differently. This is the central problem of evaluating an agent when there is no right answer. The usual escape is a rubric, which moves the problem to choosing metrics that work without ground truth rather than solving it.

The final answer hides the path

An agent that reached the correct refund amount after calling issue_refund twice and delete_order once passes an outcome test. Failures also compound: if a flow has eight steps that each succeed 95 percent of the time, the whole run succeeds about 66 percent of the time when the steps are independent, and a final-output test cannot say which step failed. Grading the path is covered in testing an agent's tool calls.

Every prompt edit is a regression with no diff

Adding one instruction on line 40 of a system prompt shifts behavior on inputs that never mention the topic, because the model weighs the whole prompt at once. Code review sees the change in text, never the change in behavior. A model upgrade is the same event at larger scale: the new version is better on average and worse on your three most important edge cases, and the provider's retirement date does not wait. Both need a suite that runs before the change lands: regression testing a prompt change and surviving a model upgrade without regressions.

The judge can be wrong in the same direction as the model

Using a model to grade a model is often the only practical option for qualitative criteria, and it inherits the model's biases: a preference for longer answers, for its own family's style, and for confident tone over correct content. Tune the prompt against the judge's scores and you optimize for the judge rather than for the customer. Keeping a judge honest is the subject of LLM-as-a-judge.

How it works

A working program has four parts: separate graders per layer, cases from real executions, a judge checked against human labels, and a gate on the version about to ship.

Four layers, graded separately

One pass or fail per case throws away the signal you most need. A run with a correct outcome and a policy violation is the one that costs money, and a run with a clumsy answer and perfect tool use is the one a prompt tweak fixes. Grade each layer with its own grader so each produces its own verdict.

Prefer the cheapest grader that captures the failure: a deterministic check costs nothing and never disagrees with itself, while a judge costs one model call per case per run and needs its criteria maintained. Split any criterion containing "and" into two graders so each returns a clear verdict.

Cases promoted from real executions

Invented cases test the failures you imagined; production cases test the failures you had. The loop is short: find a failed conversation in the logs, freeze it at the turn where things went wrong, write down the intended behavior (never the current wrong output), and add it to the suite before you fix the cause. Then confirm the case fails on the current version and passes on the fix. What to strip and what to exclude on the way in is covered in building an eval set from production traffic.

A case needs three things regardless of tooling: an input (a message, a set of variables, or a seeded conversation), a statement of intended behavior, and graders across the four layers. The shape below is deliberately tool-neutral.

{
  "name": "refund request uses the refund tool, never the delete tool",
  "input": {
    "messages": [{ "role": "user", "content": "I want a refund for order 123" }]
  },
  "expect": [
    { "layer": "tool-use", "calledTool": "issue_refund", "times": 1 },
    { "layer": "tool-use", "notCalledTool": "delete_order" },
    { "layer": "tool-use", "maxToolCalls": 3 },
    { "layer": "policy", "judge": "Confirms the order number before issuing the refund.", "severity": "gate" },
    { "layer": "outcome", "judge": "The reply tells the customer when to expect the refund.", "severity": "soft" }
  ]
}

Exclude infrastructure failures such as rate limits and provider outages, which are not the agent's fault, and remove personal data before the case is saved, because an eval set is copied and read far more widely than a log.

The judge validated against your own labels

Before an LLM judge decides anything, label 20 to 30 runs by hand as pass or fail, run the judge on the same runs, and compare. Frequent disagreement means the criteria are ambiguous; rewrite them and compare again. After that, keep recording agreement on live verdicts, one thumbs up or down each, so the fraction sits next to the score it qualifies.

Suites gate the version about to ship

An eval that runs when someone remembers is a report. An eval that runs in CI against the exact version about to be published, and returns a non-zero exit code on a failed gate, is a release control. Make correctness, tool-use and policy graders gates; mark subjective, latency and cost graders soft so they are reported without blocking a merge until you decide to promote them.

Pin one run as the baseline so every later run is a diff against a known state rather than an absolute number. Where this sits in a wider release process, including who owns the failure when the suite goes red, is laid out in a QA process for AI features.

What changes when the agent is customer-facing and multi-tenant

The techniques above were worked out for an agent one team runs and reads. An agent embedded in a product and used by many customers changes the program in five places.

Volume removes the human reader. An internal tool has a few hundred conversations a week that someone skims; a product feature has a few hundred an hour that nobody does. The suite becomes the only systematic review the agent gets, so its coverage has to be checked rather than assumed: which tools and which clauses of the system prompt have a case or grader.

Configuration multiplies the versions under test. Tenant A has a system prompt addendum about its returns window, tenant B has an extra tool, tenant C is pinned to a cheaper model, and a suite that passes against the default configuration says nothing about tenant B. Cases need to carry the tenant context they were captured under, and a model upgrade becomes a fleet event: one provider retirement, run against every configuration that depends on it.

Feedback becomes a case source. A thumbs-down or a low satisfaction rating marks the exact turn where the agent lost the customer, and the conversation up to that turn is a regression case waiting for review. Nothing should enter the suite unreviewed, since a customer can be wrong, but a queue of proposals from real customers beats a brainstorm.

Privacy constraints tighten. Captured inputs now contain other people's order numbers, addresses and medical questions. Redaction has to happen before the case is saved, and the eval set needs the same access controls as production data.

The policy layer carries the money. What the agent may promise, the refund it may issue without a human, the claim it may make about a delivery date: for an internal tool these are embarrassments, for a customer-facing one they are liabilities. Policy graders become gates, and the approval step before a consequential tool call gets its own case asserting that the agent asked rather than acted.

Where Runtype fits

Runtype turns runs an agent has already produced into the cases that gate its next version.

Start by sending traces. Any OpenTelemetry-instrumented agent exports OTLP/HTTP to https://api.runtype.com/v1/otel and appears in the Runs view with its trace tree, logs and token usage. A run whose spans carry the GenAI content attributes has a transcript that can be captured as an eval case, so the suite grows from the failures you had. Point exactly one instrumentation at the endpoint, since two doubles tokens and cost.

Register the agent next. An external agent whose endpoint speaks A2A can be an eval-suite target, so a suite re-runs it rather than reading its history; cases that replay recorded tool activity are skipped, and a runtype-stream endpoint cannot be re-run by a suite. Registration also makes it testable from the dashboard and usable as a product capability. An MCP surface points the other way, exposing flows, agents, records and tools to your existing loop.

Rebuild one capability natively once the harvested suite can prove parity for it, and the gate comes with it. runtype eval run runs every case against the target in CI and exits 0 when every gate passes, 1 on a failed gate and 2 on a configuration error. One run can be pinned as the baseline.

The four layers map to three grader kinds. Deterministic checks inspect the output; trace graders inspect recorded behavior with builders such as calledTool, notCalledTool, toolOrder and maxToolCalls at no model cost; judges receive the recorded steps and tool calls beside the final output and return pass, fail or insufficient evidence with reasoning linked to the step it cites. Each grader is a gate by default and can be marked soft. Each AI verdict carries a thumbs up or down, and the suite reports judge trust as a fraction such as "Agrees with you 7 of 9 times".

Cases arrive from executions: a failed conversation frozen at a fork point, a flow test saved from the debugger, or POST /v1/eval/suites/{id}/cases/from-execution. A review queue collects proposals from a thumbs-down, a low rating, generated edge cases and coverage gaps, and the coverage panel reports which target tools and instruction clauses have no case. Identity is a declared property of each resource, a tenancy strategy of internal, tenant-isolated or end-user-isolated evaluated before execution, and every trace and cost figure is filed under the tenant and end user it ran for, so a failure only one customer sees is findable while it is still a single run.

Setup starts at what are evals; the CI path is manage evals as code. The suite decides whether the next version reaches your customers.