How to evaluate an AI agent when there is no single right answer

Grade the properties you care about instead of the exact string. Split every run into four layers and score them separately: whether it reached the right outcome, whether it called the right tools in a sane order, whether the content is grounded in what it retrieved, and whether it stayed inside policy. Deterministic checks run on every commit, model-graded rubrics run on a slower cadence, and the combined verdict says which layer failed.

An agent that summarizes a ticket, drafts a plan or answers a support question has a wide band of acceptable outputs and a narrow band of unacceptable ones. An equality assertion describes one point in that band, so the suite either fails on harmless rewording or gets deleted within a month. What usually replaces it is a spreadsheet somebody skims until the novelty wears off, a review process with no threshold and no memory of last week.

How do I test an AI agent when the output is different every time

Assert on invariants rather than on text. Two runs of a refund-policy question can differ in every word and still both be correct, but they cannot both cite a policy section that does not exist, and they cannot both skip the account lookup. Those are the properties worth encoding.

Keeping them separate matters because the fixes differ. A grounding failure is a retrieval or prompt problem, a policy failure is usually a system-prompt or guardrail problem, and a trajectory failure is a tool description or tool-selection problem, covered in more depth in testing agent tool calls. A single blended score of 0.71 tells you none of that.

Step 1: write the failure modes down before you write the rubric

Collect twenty to thirty real transcripts and mark what went wrong in each one, in the words a support lead would use. You will get a short list: invented a policy number, promised a refund it cannot authorize, answered from memory without searching, looped on the same search three times, correct but four paragraphs longer than anyone will read.

That list is the rubric's contents. A rubric written before the failures is a list of virtues (helpful, accurate, concise) that a model grader will score high on almost everything, because virtues have no edges. Pull the starting transcripts from production traffic rather than from a synthetic set; the mechanics are in building an eval set from production.

Step 2: run the deterministic checks first

Anything you can decide with code should never reach a model grader. These are cheap enough to run on every commit and they never drift:

• Structural: the response parses as the schema you asked for, required fields are present, cited identifiers exist in your database.
• Trajectory: the account lookup happened before the refund tool; no tool was called more than N times; no tool was called with a null tenant id.
• Policy patterns: no phone numbers in the output, no promise verbs from a fixed list, refusal phrasing present when the input matches a category that must be refused.
• Cost and shape: the run finished inside the turn budget, the output is under the length ceiling.

Each of these is a decision a reviewer could re-derive from the run log by hand, which is why they hold their meaning across model versions. They are also the only checks you can attach to a pull request without a token bill, which is what keeps them running.

Step 3: write a rubric with anchors, not adjectives

A model-graded criterion needs one dimension, a small integer scale, and a written description of what each score looks like. Anchors are what stop two runs of the same grader from disagreeing.

criterion: grounding
scale: 0-3
inputs: [agent_output, retrieved_passages]
anchors:
  3: Every factual claim maps to a specific retrieved passage. Numbers, dates
     and policy identifiers match the source exactly.
  2: All claims are supported, but one number, date or identifier is
     paraphrased loosely enough that a reader could misquote it.
  1: At least one claim is unsupported by any retrieved passage, though
     nothing contradicts the passages.
  0: A claim contradicts a retrieved passage, or a cited identifier does
     not appear in the passages at all.
output: { score: integer, offending_claim: string|null, rationale: string }

Three rules make anchors work. Keep the scale short, because a grader cannot distinguish seven levels reliably. Give each criterion its own call rather than asking one prompt for five scores at once, since a single low dimension drags the others down. Require the grader to quote the offending span before it gives a score, so a low score comes with evidence a human can check in seconds.

Step 4: choose reference-free or pairwise grading

Reference-free grading scores one output against the rubric with no gold answer. Use it for grounding and policy, where the standard lives in the retrieved passages and the rules rather than in an ideal response. It gives you an absolute number you can put a threshold on, which is what a release gate needs.

Pairwise grading shows the judge two candidate outputs for the same input and asks which better satisfies the criterion, with a tie allowed. Use it for outcome and helpfulness, where a rubric is hard to write but a preference is easy to state. Judges are more consistent choosing between two candidates than assigning an absolute number to one.

{
  "input": "Customer asks whether a 45-day-old order qualifies for return.",
  "criterion": "outcome: gives the customer a correct, actionable next step",
  "candidate_a": "...output from the current prompt...",
  "candidate_b": "...output from the proposed prompt...",
  "instruction": "Answer with A, B, or TIE, then quote the sentence that decided it.",
  "swap_order_and_repeat": true
}

Two mechanics matter more than the prompt wording. Judges favor whichever candidate appears first, so run every comparison twice with the positions swapped and count a pair as a tie when the two runs disagree. Report a win rate over the whole set with the tie count beside it, and treat a small margin on fifty cases as noise rather than a result.

Step 5: check the grader against human labels

A model grader is a measuring instrument, and an uncalibrated instrument produces confident numbers about nothing. Calibrate it once, and again after any change to the rubric, judge model or judge prompt.

1. Sample a hundred or more cases from your eval set, weighted so failures are well represented rather than matching production's pass rate.
2. Have two people label each one against the same written rubric, independently.
3. Compute human-to-human agreement first. That number is the ceiling; if your two labelers agree only 70 percent of the time, the rubric is ambiguous and no grader will beat it.
4. Compute grader-to-human agreement with Cohen's kappa or, on a 0-3 scale, exact-match plus within-one-point rates.
5. Read the disagreements. They cluster, and the cluster usually names a missing anchor rather than a defective judge.

Fix the rubric and repeat before you touch the judge model. Hold out a slice of the labeled cases so you have an untouched set to test the revised rubric against. When agreement stalls below the human ceiling on a criterion, that criterion is a candidate for human review rather than automation. The wider set of ways to measure quality without a gold answer is in LLM evaluation metrics without ground truth, and the failure modes of judge models themselves are in LLM as a judge.

Step 6: combine the layers without hiding which one failed

Report a per-layer verdict, then one gate built from them. A reasonable shape: policy and trajectory are hard gates that fail the case outright, grounding needs a mean score at or above 2.5 with no case scoring 0, and outcome win rate must not drop against the previous published version by more than the pairwise noise margin you measured on that set.

Store the per-layer scores with the run, never only the aggregate. When a suite goes from 94 percent to 88 percent, the useful question is which layer moved, and an aggregate cannot answer it. Every case that fails in production becomes a new case in the set, with the layer it broke recorded, which is what turns the suite into something that accumulates rather than something that decays. The surrounding practice, including how often to run each layer and what to gate a release on, is covered in AI agent evals.

Where this gets easier

Most of the work above is plumbing: keeping the trace, the tool call log, the retrieved passages and the four scores attached to the same run so a failure points at a step rather than at a number. Runtype's eval suites grade all four layers on the same run and keep the per-step trace attached to each score, with cases promoted from a recorded execution, human review of individual judge scores, coverage reporting, and run-to-run and record-level comparison so a regression names the step that caused it.