FeatureJuly 4, 20261 min read
Turn real runs into eval cases
Capture any real agent execution as a test case: fork at a point in the conversation, replay with recorded tool results, and let user feedback propose regression cases automatically.
evalstracesagentstesting
The best test cases come from real usage. You can now turn actual executions, and even end-user feedback, into eval cases without writing them by hand.
- Capture from a trace. An "Add to eval" button on agent log traces forks a real run into a test case: pick the fork point, and the conversation plus every recorded tool result is frozen in. Cases show a Replayable or Partial replay badge.
- Save as test case. After a test run in the flow Test tab, one click saves the run into the flow's default eval suite.
- Next-step replay. A captured case runs its frozen history for exactly one assistant action. The action is recorded, never executed, and graders judge it. Run results show the recorded action beside this run's so divergence is obvious.
- Continue-mode replay. A recorded tool executor serves captured tool results by name with no real I/O, so a replay can run past the fork and grade the final output.
- Feedback becomes proposals. A thumbs-down or low survey score proposes a regression case; a thumbs-up proposes a golden one. You can also rate each AI-grader verdict, and the suite shows how often the judge agrees with you.
- A governed proposal queue. Machine-generated cases land in a review queue; nothing enters a suite without your accept. A coverage meter shows which tools your suite exercises.