How to build your first eval set from real production conversations

Mine the eval set out of production traces rather than writing cases from imagination. Take a stratified sample across four outcomes, clean success, human escalation, user retry, and hard error, convert each sampled trace into a case with inputs and criteria written for that case, and adopt a standing rule that every reported bug becomes a case before anyone fixes it.

A first set of thirty to sixty cases built this way is more useful than three hundred invented ones, and it takes an afternoon rather than a sprint.

The cases you invent test the failures you already imagined

An engineer sitting down to write a hundred test cases produces a hundred variations on the inputs they have in their head. Those inputs are the happy path, plus the two or three edge cases that came up in review. The failures that actually reach support are different in kind: a user who answers a clarifying question with a screenshot, a tenant whose knowledge base contains two contradictory refund policies, a tool that returns an empty array and gets read by the model as a permission error.

Production has all of them already, attached to the exact configuration that produced them. What most teams lack is a way to get them out, which is a sampling and consent problem more than a tooling one. The category background is at AI agent evals.

Where do I get test cases for my LLM app

From the executions you have already run. You need three things in place before the loop below works: traces that retain the full input and output of each step along with tool calls and their results, at least two or three weeks of them, and some signal for how each conversation ended. If your traces record only the final answer, fix that first, because a case built from a final answer alone cannot assert anything about tool behavior. What to capture is covered in what to log from an LLM app.

Build the set in six steps

1. Decide what you are allowed to sample

Copying a customer conversation into an eval suite is a new purpose for that data, and your data processing agreement either covers secondary use for testing, excludes it, or says nothing. Silence is not permission. Get the list of tenants in scope from whoever owns those contracts, and encode it as a filter in the query that builds the sampling frame, not as a review step afterward.

Exclude two more categories at the same point: anything under legal hold or an active deletion request, and your own staff traffic, which is disproportionately weird and will skew every stratum.

2. Stratify by outcome, not by volume

A random sample of production traffic is mostly the boring majority case, so it tells you almost nothing about the failures. Split the frame into four strata and take a quota from each:

Retry detection is the one worth building carefully. A cheap version matches a short list of phrases ("no", "that's not", "I meant", "still", "again"), and it will have false positives. A better version compares the two user messages for overlap and flags the pair when the second repeats a noun phrase from the first. Either way, review the flagged pairs by hand for the first batch, because the detector is also a signal you will want in production later.

Add a fifth stratum once the first set is running: turns whose tool-call count sits above your 95th percentile. Those are where loops and retry storms live, and they rarely appear in the other four.

3. Redact without destroying the case

Run redaction where the trace is stored, before anything is copied into an eval suite, and keep the policy identical to the one your logs already use so a case cannot contain more than a log line would. The mechanics of that are in PII redaction for LLM logs.

The failure mode specific to evals is over-redaction. Replacing every digit sequence with a placeholder turns an order-lookup case into a case about missing information, and it will pass, because the agent correctly asks for the order number it can no longer see. A case that passes for the wrong reason is worse than a missing case. Substitute stable synthetic values instead: order 10024 becomes order 10024 everywhere in that case, a real email becomes ada@example.test, and the shape, length and format stay intact so format-sensitive tool calls still behave.

Keep the source execution id on the case. When a deletion request arrives for the underlying conversation, that id is how you find the derived cases and remove them too.

4. Keep the eval set inside the tenant boundary

An eval suite is a cross-tenant surface by construction. The moment tenant A's conversation lands in a shared suite, every engineer who can run that suite reads A's content, and so does whatever judge model scores it. Treat the suite as tenant data with its own access list rather than as test fixtures. The wider checklist is at tenant isolation for AI agents.

Two rules keep the set honest. Never merge context from two tenants into one case, because the resulting conversation never happened and the agent's behavior on it means nothing. And record the tenant configuration the case was captured under, since a case sampled from a tenant with a custom knowledge base and a restricted tool set only reproduces under that configuration.

5. Cut a recorded execution into a case

Pick the turn under test, then take everything up to and including the last user message as input. The case needs the messages, the resolved variables, and the tenant configuration identified by version. Freeze the tool responses if you are testing the model's reasoning; leave them live if you are testing the integration.

{
  "id": "case-0142",
  "source_execution_id": "exec_01j9r2fk4",
  "stratum": "user_retry",
  "tenant": "acct_4412",
  "tool_mode": "recorded",
  "added_at": "2026-08-19",
  "input": {
    "messages": [
      { "role": "user", "content": "where is order 10024?" },
      { "role": "assistant", "content": "I can look that up. What is your email?" },
      { "role": "user", "content": "i already gave it to you" }
    ],
    "variables": { "locale": "en-GB", "plan": "growth", "config_version": "17" }
  },
  "criteria": [
    { "type": "assert", "rule": "calls lookup_order with order_id 10024" },
    { "type": "assert", "rule": "does not ask for the email again" },
    { "type": "judge", "rule": "acknowledges that the email was already provided" }
  ]
}

One rule about expected output: when the recorded execution was correct, the recorded answer is a fine reference. When it was wrong, write what should have happened in words and never paste the wrong output anywhere the scorer can read it as a target.

6. Write criteria per case, and say who wrote them

Global metrics like helpfulness produce a number that moves for reasons nobody can trace. Criteria attached to the individual case are checkable. Each case gets what must happen, what must never happen, and a note about which criteria are mechanical assertions against tool calls or JSON fields and which need a judge. Approaches for cases with no single correct answer are in LLM evaluation metrics without ground truth.

Have two people label the first twenty cases independently and compare. Disagreement means the criterion is ambiguous, which is a defect in the criterion rather than in either labeler. Rewrite until two people reading the same case predict the same verdict.

How big the set needs to be

Smaller than most teams assume. Thirty to sixty cases with eight or more per stratum is enough for a first set, because the number you are watching is a change in pass rate between runs, and at forty cases a single broken case moves it by two and a half points. Adding two hundred more cases of the same kind buys resolution you do not need and a maintenance cost you will not pay.

Before trusting any delta, measure the noise floor: run the unchanged set twice and record how much the pass rate moves on its own. Sampling temperature and judge variance both contribute, and a set whose own spread is five points cannot detect a three-point regression. Running the set against a change is covered in prompt change regression testing.

Keeping the set from rotting

The standing rule does most of the work: every reported bug becomes a case before the fix is merged, reproduced from the execution that produced it rather than from the bug report's paraphrase. This is the habit that decides whether the set stays alive, and it fails for a mundane reason: when copying a trace into a case by hand takes longer than the fix itself, the rule gets skipped under deadline every time.

Three maintenance habits handle the rest. Re-sample every quarter, because the input distribution moves as the product changes and your strata quietly stop matching reality. Retire cases explicitly with a date and a reason when the behavior they pin is no longer wanted, instead of deleting them quietly. And review cases that have passed forty consecutive runs, since a case that has never once failed is usually asserting something the system cannot get wrong.

Where this gets easier

Runtype can promote any recorded execution into an eval case in one step, which is what turns "every bug becomes a case" from a chore into a habit. Because the execution trace already carries per-step input and output, tool calls with their arguments and results, and the tenant and end-user identity the run happened under, the case arrives with the context a hand-built fixture usually loses. Suites run with judge scoring and human review of individual scores, coverage reporting shows which parts of the agent no case exercises, and run-to-run comparison gives you the delta against a previous run. Redaction policy and tenancy strategy are set per product, surface or agent and resolved at dispatch, so the eval data inherits the same boundaries the production traffic ran under.