LLM-as-a-judge: when to trust it, and how to check the judge

A model grader earns trust one decision at a time. Measure its agreement with human labels on that exact decision, on your own data, before it gates anything, then publish the agreement figure beside every score it produces. Keep deterministic checks for every property a rule can express, and spend judge calls on the properties no rule can reach.

Model grading is the only affordable way to score open-ended output at volume. A team shipping a support assistant produces more transcripts in a day than a reviewer reads in a month, and a rubric applied by a model costs a fraction of a cent per case. The grader is also a model, and it carries preferences about length, confidence, formatting and the family of models it came from.

Two habits form around that fact. One team decides model grading is unscientific and returns to a spreadsheet nobody fills in after the third week. Another wires a judge into CI, watches the average score, and never checks whether that average tracks anything a person would agree with. Both arrive at the same amount of evidence about quality.

Is LLM as a judge reliable enough to gate a release

Reliability is a property of the task you hand the judge. Comparing an answer against a passage supplied in the same prompt is a reading-comprehension task, and models are consistent at it. Scoring the overall helpfulness of an answer from 1 to 5 with no rubric is a preference task, and preference is where the biases live.

The release question has a conditional answer. A judge can gate on a task in the first two rows once you have measured its agreement on that task, pinned the judge model version and rubric, and written down what happens when the gate fires. It cannot gate on a bare quality score, and it should not be the only thing between a change and a customer. Where a judged score fits among the other numbers a team tracks is covered in AI agent evals and in what to measure with no ground truth.

What makes a judge score stable

A rubric that produces the same verdict twice has a few properties, and most unstable judges are missing several of them at once.

• One property per call. A single prompt asking for groundedness, tone and completeness produces one blended number that moves for reasons you cannot separate.
• Binary or three-point ordinal levels, each defined by observable evidence rather than by an adjective. "Every factual claim appears in the supplied text" is checkable, "the answer is high quality" is not.
• The evidence lives in the prompt. A judge asked whether a claim is true reasons from training data; a judge asked whether a claim appears in a supplied passage reads.
• Reasoning before verdict, and a quoted span as part of the output. The span is what lets a reviewer disagree with a specific score instead of with the judge in general.
• A pinned model version and a fixed temperature. Determinism is still not guaranteed, so run the labeled set three times when you first establish a baseline and record the spread.

A rubric worth versioning looks closer to a schema than to a paragraph:

{
  "property": "grounded",
  "inputs": ["answer", "retrieved_context"],
  "levels": [
    { "value": "supported", "definition": "Every factual claim appears in retrieved_context, paraphrase allowed." },
    { "value": "partly_supported", "definition": "At least one claim is absent from retrieved_context, and none contradict it." },
    { "value": "contradicted", "definition": "At least one claim conflicts with a statement in retrieved_context." }
  ],
  "output": { "value": "one level id", "evidence": "quoted span from retrieved_context, or empty" },
  "judge_model": "pinned-dated-version",
  "rubric_version": 4
}

Treat that file the way you treat application code. Any edit to a level definition invalidates the calibration behind it, which is why the version number belongs in the record of every score.

Position, verbosity and self-preference bias

Three biases are well enough documented that you should assume all three are present until you have tested for them.

The padding control is the cheapest of the three to run, and it usually fires first. Take twenty answers the judge already scored, append a paragraph of correct but unrelated background to each, and re-score. A rubric that survives it is scoring content; a rubric whose scores climb is scoring effort.

Calibrating against human labels

Calibration is four steps, and the first one decides whether the resulting number means anything.

1. Sample deliberately. Take 50 to 100 production cases for the exact property you are judging, stratified so that failures make up a third to a half of the set. A uniform random sample of a healthy system is mostly passes, and a judge that always says pass scores well on it.
2. Label twice. Two people label independently against the same rubric, then reconcile the disagreements. Human agreement on that set is your ceiling: if two reviewers agree on 85 of 100 cases, no judge should be asked to beat 85.
3. Run the judge cold. Same prompt, same model version, same temperature as production, on the same cases, with no access to the human labels.
4. Report three numbers, not one. Raw agreement, a chance-corrected figure such as Cohen's kappa, and recall on the failure class.

Take a set of 80 cases where reviewers found 30 real failures:

Raw agreement is 67 of 80, or 84 percent. Expected agreement by chance is (25 × 30 + 55 × 50) / 6400, which is 0.55, so Cohen's kappa is (0.84 - 0.55) / (1 - 0.55), about 0.64. Recall on failures is 21 of 30, or 70 percent.

Eighty-four percent reads like a pass in a review meeting. The same judge lets three of every ten real failures through the gate, and the chance-corrected figure says a good share of that 84 comes from both parties agreeing on easy passes. Publish all three numbers next to the score on any dashboard, and re-run the labeled set whenever the judge model, the prompt, the rubric or the traffic being scored changes. Gating a prompt change on scores from an uncalibrated judge is covered further in regression testing prompt changes.

When a deterministic check is the better tool

Every property a rule can express should be a rule. Schema validity of a structured response, citation identifiers present in the retrieved set, a required disclaimer string, a forbidden phrase, whether the account lookup tool was called at all, whether an argument matched the customer id in the request, a number inside a tolerance: none of these need a model, and a model makes each of them slower, costlier and noisier.

Order matters as much as the choice. Run the deterministic layer first and send only the survivors to the judge, which cuts judge spend and gives you failures attributed to a named rule rather than to a score. The layering of deterministic and judged checks across outcome, trajectory, grounding and policy is worked through in evaluating an agent with no single right answer.

A judge earns its place where the property requires reading: whether an answer is grounded in what was retrieved, whether a refusal fits a policy in an unusual situation, whether one of two acceptable answers is better, whether a tone instruction was followed. Those are the cases a regular expression cannot state, and they are the ones worth paying a model to read.

Which decisions a judge can make on its own

Sort the decisions by what a wrong verdict costs, and let the judge own the cheap end.

• On its own: ranking candidate prompts during development, triaging which sessions a person should read, tracking a trend on a fixed sample with a fixed rubric, flagging outliers for review.
• With a person reading a sample: a release gate, anything a customer sees, any batch where scores cluster near the threshold, and the failure class of the calibration set at each release.
• Never alone: blocking or allowing on safety and compliance grounds, choosing between two model families with a judge from one of them, publishing a score to a customer as an accuracy claim, and editing the rubric that defines the score.

Scores near the threshold deserve the most caution: a change that moves a gated score from 0.79 to 0.81 sits inside the run-to-run spread of most judges, so a promotion rule that reads "ship on any improvement" ships changes at random.

Where this gets easier

Runtype runs judge scoring inside eval suites and keeps human review of individual scores in the same place, so a score a reviewer disagrees with becomes a record attached to the case rather than a comment in a spreadsheet. Cases can be promoted from a recorded execution, which is how a calibration set gets built out of real traffic instead of invented examples, and coverage reporting shows which properties actually have cases behind them. Run-to-run and record-level comparison then reads a change against the same set, so the agreement between the judge and your reviewers is something you compute from the run rather than assume. Background on that model is at what are evals.