FeatureJuly 4, 20261 min read

Smarter AI graders: trajectory judging, focused criteria, and custom judges

AI graders now judge the whole run with evidence-backed verdicts, route to a judge from a different model family by default, and can be split into focused checks or replaced with your own judge flow.

evalsai-gradersdashboardtesting

AI graders got a major overhaul. They now evaluate how an agent actually behaved, not just its final answer, and every part of the judge is configurable.

Trajectory judging. The AI grader reads a transcript of the full run (messages and tool calls) and returns a binary pass or fail verdict with cited evidence. Evidence chips deep-link into the trace, and a judge that lacks evidence declines with a warning instead of guessing.
Family-aware judge routing. When a grader doesn't pin a model, the default judge is swapped for a model from a different family than the one being evaluated, reducing LLM self-preference bias.
Split into focused checks. Decompose one fuzzy criterion into independently judgeable sub-checks, reviewable before you accept them, in the new dashboard grader editor or via evals.decomposeCriteria in the SDK.
Custom judge flows. Set judgeFlowId on an AI grader to use one of your own flows as the judge instead of the platform prompt.
Grader severity. Mark any grader soft to warn instead of fail; strict runs still enforce it.
A real home for evals. Each suite gets a detail page with a score-over-time chart, run history, and baseline pinning, plus a Score column in the comparison view.