Faster, durable, context-aware evals

A wave of eval-engine upgrades landed this week, aimed at three things: runs that finish, judges that grade with full context, and suites that can express real test plans.

Run mechanics

• Parallel judge scoring. AI judges (platform judges and custom judge flows) now score through a bounded concurrency pool, so large suites finish up to 4x faster and stop timing out the synchronous run endpoint. Results are byte-identical to sequential order.
• Durable sync runs. A synchronous eval run appears in run history the moment it starts, and the whole run completes server-side even if your client times out or aborts. Failed runs finalize as failed instead of stranding in a processing state.

What the judge sees

• Seeded conversations. For conversation-seeded cases, the judge transcript now opens with the seeded dialogue, so you no longer have to restate conversation context in expected.facts. Custom judge flows get this automatically.
• Deterministic checks as ground truth. Each case's check outcomes (tool called, step ran, run completed, output contains) are rendered into the judge transcript as settled facts the judge must not re-litigate, so structural questions get decided once, deterministically.

What a suite can express

• Per-case graders. Cases can carry their own graders array that replaces the suite's graders for that case, on the API, MCP tools, and both SDKs. In defineEval, suite-level graders merge with each case's expect list, and the server decomposes the result automatically; the old "every case must share one grader set" error is gone.
• Hand-written tool mocks. input.toolMocks is now authorable on any case, with no captured run required, so you can evaluate agents that call local or expensive tools. isError: true replays a call as a failure.
• Conversation-seeded cases on the batch path. Multi-turn cases now run on the durable batch path, lifting the 50-case ceiling that previously applied to them.

Details and worked examples are in the evals-as-code guide.