SDKJune 26, 20261 min read

Code-colocated evals: defineEval in the TypeScript SDK

Define evals next to your flows and agents with defineEval, converge them with evals.ensure, and gate CI with the runtype eval run command. Trace graders assert on tool calls, step order, and cost.

evalssdkclici

Evals now live in your repo, right next to the flows and agents they test. Author them in TypeScript, converge them at deploy time, and fail CI when quality regresses.

defineEval. A pure, typed constructor for eval suites: target, cases, and graders, with builders like contains, matchesExpected, jsonField, and judge for AI grading.
evals.ensure. Hash-first, idempotent converge for suites, the eval sibling of flows.ensure. Unchanged suites are a no-op; evals.pull reads the platform's current definition back.
runtype eval run. The CLI discovers *.eval.ts files, runs each suite synchronously against its target, and maps results to exit codes, with --junit output for CI. Backed by the new synchronous /eval/run endpoint.
Trace graders. Deterministic, free assertions over how the run behaved: calledTool, toolOrder, ranStep, stepOrder, completed, maxToolCalls, and cost.
Inline evals: on flows. Attach suites directly to defineFlow; they converge with the flow, defaulting to it as the target. Graders can chain .gate() or .soft(), and --strict promotes soft misses to failures.

One CI step, and an eval regression stops in review instead of production.