What to measure when your AI feature has no ground truth
A working set of LLM evaluation metrics you can compute without labels, how each one is calculated from traces and product events, and what each misses.
Measure outcomes you can observe without an answer key: whether the task finished, whether claims were supported by the sources the agent actually retrieved, whether tool calls succeeded, whether refusals matched policy, how often a human took over, plus latency and cost per resolved task. Each number on the dashboard should trigger exactly one decision.
Why BLEU, ROUGE and a generic quality score move for no reason
BLEU and ROUGE were built for tasks that ship with reference texts. They count n-gram overlap between a candidate string and one or more references, which works when a human translation or a reference summary exists for every input. A support answer written this morning has no reference, so a correct answer phrased differently from last week's scores low, and a fluent answer that repeats the customer's wording back at them scores well.
The generic judge score has a different failure. A prompt that asks a model to rate an answer from 1 to 5 with no rubric produces a number that responds to answer length, to the order options were shown in, to the judge model's version, and to whatever the last person changed in the judge prompt. When it drops from 4.1 to 3.9 in a review, nobody can name the change that caused it, so the meeting moves on and the metric quietly stops being used.
Apply one test to every candidate metric before it goes on a dashboard: if this number moved 10 percent on Tuesday, what would we do differently before Friday? A metric with no answer to that question is a decoration. The set below exists because each entry has an answer, and because each can be computed from data a running system already produces. The category background sits in AI agent evals; the harder question of how to judge one open-ended answer fairly is evaluating an agent with no single right answer.
What metrics should I track for an LLM feature in production
Seven numbers cover most customer-facing features. Five come from traces and product events at zero marginal cost, and two need a judge model over a sample. Read the last column before adopting any of them, because a metric whose blind spot matches your actual failure mode is worse than no metric.
| Metric | What it measures | How it is computed without labels | What it misses |
|---|---|---|---|
| Task completion rate | Share of sessions that reached the terminal signal the feature exists to produce | Product event: ticket closed and not reopened within 48 hours, order submitted, draft accepted without edit, no rephrase of the same question within one session | A user who got a wrong answer, believed it, and left satisfied. Sessions where the terminal event is recorded by another system that lags |
| Grounded-claim rate | Share of factual statements in an answer that are supported by the sources retrieved for that same turn | Judge model receives the answer plus the exact retrieved context, and marks each claim supported, contradicted or unsupported. No gold answer needed, since the check is support against a given text | Whether the source itself is correct or current. Omissions, because an answer that leaves out the important caveat is fully grounded |
| Tool-call precision | Share of tool calls that were well-formed, returned without error, and whose result reached the answer | Trace counters: arguments failing schema validation, non-2xx and timeout results, identical repeated calls in one turn, results never referenced in the final message | Tool recall. A turn that should have called the refund lookup and did not is invisible here, and reads as a clean turn |
| Refusal correctness | Of turns where the agent declined, the share policy says it should have declined, and the inverse | Judge model with the policy text on all refusals plus a matched sample of answered turns. Refusals are usually rare enough to score every one | Soft refusals: a hedged non-answer that never uses refusal language, so it is counted as an answer |
| Escalation rate | Share of sessions handed to a human, or where the user asked for one | Product event on handoff, plus a phrase match on the user turn for a request to reach a person | The reason. A correct escalation on a policy-restricted request and a failed one look identical until someone reads the transcripts |
| Latency, p95 and p99 | Time to first token and time to the final answer, at the tail | Timestamps already on the trace, bucketed by surface and by whether the turn used tools | Perceived wait during a stall mid-stream. Turns that errored and were excluded from the denominator, which is where the slow tail usually hides |
| Cost per resolved task | Spend divided by tasks that hit the completion signal | Token cost of every call in the session, plus tool vendor spend, divided by completed sessions rather than by all sessions | Attribution of shared spend such as retrieval indexing. A cheap failure looks better than an expensive success |
1. Define the unit of work before defining any metric
Pick the unit the feature is answerable for: a session, a ticket, a document processed, a scheduled run. Every metric in the set then uses that unit as its denominator, which is what makes them comparable. Teams that skip this end up with completion measured per session, groundedness measured per answer and cost measured per API call, and the three cannot be put in one sentence.
2. Take outcome signals from product events, not from the model
Task completion and escalation are facts about your product, so read them from the systems that already record them. A ticket that closed and stayed closed, a form that submitted, a draft accepted without an edit: these are events with timestamps and identity attached. Asking a model whether the user seemed satisfied reintroduces the judge into a place where a plain event exists.
3. Score groundedness against retrieved context, not against the world
Give the judge the answer and the exact context string the agent had at that turn, and ask it to classify each factual claim as supported, contradicted or unsupported by that text. The task is comparison between two passages, which models do far more consistently than open-ended correctness. Report the claim-level rate and the share of answers with at least one unsupported claim, because a single fabricated policy number in an otherwise correct paragraph is the failure customers escalate.
4. Take behavior metrics off the trace
Tool-call precision and its components are counters over structured data, so they need no judge at all. The three that predict user-visible failure are schema validation errors, repeated identical calls within one turn, and tool results that never appear in the final answer. The last one catches an agent that retrieved the right record and then ignored it, which reads as a confident wrong answer to the customer and as a healthy turn on a latency chart.
5. Give every metric a threshold and a named action
Write the action next to the number, in the same document: grounded-claim rate below 0.95 for a week freezes prompt changes and opens a review of retrieval; tool-call precision dropping on one tool pages the team that owns that integration; escalation rate rising with completion flat means the agent is punting rather than failing. Without this column, a dashboard becomes a weather report. Failures that never move any of these numbers are a separate problem, covered in finding silent failures in an AI feature.
6. Version the definitions and record the changes
Store each metric definition as text in the repository next to the code that computes it, and change it in a commit. When completion rate jumps 6 points, the first question is whether the definition changed, and the second is whether traffic mix changed. A metric whose definition drifts without a record is a metric that will lose an argument in a review.
7. Report with denominators and traffic mix attached
Publish the rate, the numerator and the denominator together, split by surface and by tenant cohort. A completion rate averaged across a chat widget and a batch job hides both. Volume belongs beside quality for the same reason, since a metric improving while usage collapses usually means the hard traffic went somewhere else, which is why adoption belongs in the same review (measuring adoption of an AI feature).
What to sample and what to measure on every run
Anything computed from a counter runs on every execution, because it costs a query rather than a model call. Completion, escalation, tool-call precision, latency and cost per resolved task all fall in that group, and running them continuously is what lets you catch a regression on the day it ships rather than at the end of a sampling window.
Judged metrics are sampled. Grounded-claim rate and refusal correctness each need a model call per scored item, so scoring every production turn doubles the cost of the feature and adds a second model whose failures you now also have to monitor. Draw a fixed-size stratified sample per week, keep the strata and the judge prompt constant between runs, and version the judge model explicitly so a provider upgrade does not arrive as a quality regression.
Two exceptions are worth the extra spend. Score every refusal, since refusals are usually a small fraction of traffic and each one is a customer who did not get an answer. Score every turn from a tenant currently in an escalation, because during an incident the sampling interval is longer than the patience of the people asking.
Where this gets easier
Most of this set is a query rather than a new instrumentation project when the platform running the agent already records the execution. Every Runtype execution stores the step trace with per-step input and output, tool calls with their arguments and results, latency, and cost per execution, record and batch, so completion signals, tool-call precision, tail latency and cost per resolved task come from data that is already there. The judged metrics run as eval suites with LLM-judge scoring and human review of individual scores, with cases promoted from recorded production executions and coverage reported across the suite. Agents built elsewhere can send OpenTelemetry traces to https://api.runtype.com/v1/otel and be scored the same way.
Frequently asked questions
- Is BLEU or ROUGE ever useful for an LLM feature?
- Rarely, and only where a reference text genuinely exists. Both score n-gram overlap against one or more reference strings, which fits machine translation and reference summarization benchmarks. A production assistant has no reference for the answer it just produced, so the score rewards paraphrase similarity rather than correctness. If you have a frozen set of answers a human wrote and approved, overlap is still a weak proxy for whether a new answer is right.
- How large a sample do I need for a judged metric?
- Large enough that the confidence interval is narrower than the change you want to detect. For a rate near 90 percent, 200 sampled sessions give roughly plus or minus 4 points at 95 percent confidence, so a 2 point weekly wobble is noise. Stratify the sample by tenant, surface and traffic type so one large customer does not dominate it, and keep the sample size fixed between runs.
- Should I report one overall quality score to executives?
- Report the set, with the denominators, and put the escalation rate and cost per resolved task at the front. A single blended score hides the trade a change actually made, because groundedness can rise while completion falls when the agent starts refusing more. If a summary number is unavoidable, define it as a fixed weighted combination, version the weights, and show the components beside it.