How to find the small share of conversations that are silently failing

Silent failures show up as behaviour rather than as errors. Look for users who rephrase the same question, abandon a thread mid-conversation, or escalate to a human, and for traces where a tool returned nothing and the model answered anyway. Sample a week of those, review them against a fixed rubric, and rank the causes by cost.

The work is a weekly loop rather than a dashboard. Eight detectors, a stratified sample of about seventy conversations, and a couple of hours of reading turn a week of traffic into a ranked list of failure classes you can fix.

Near-zero error rate, unhappy users

Every request returned 200, every span closed, latency sits where it always sits. A model that answers the wrong question confidently produces the same telemetry as a model that answers correctly, because the failure is in the meaning of the text and nothing in the stack reads meaning.

A concrete version: a tenant's knowledge base was never indexed after onboarding, so the retrieval tool returns an empty array on every call. The model treats an empty result as "nothing relevant exists" and answers from its general knowledge, in the same tone it uses when the retrieval worked. The user reads a plausible, wrong answer about that tenant's own refund policy and closes the tab.

Thumbs-down controls capture the users who care enough to click, which is a small and self-selected slice of the ones who were failed. One of the few published measurements of that slice comes from a 2025 Google study of an internal developer assistant, Reading Between the Lines: across 372 developers over nine weeks, thumbs-up and thumbs-down ratings landed on about 0.6 percent of the roughly 36,000 turns. That is one product with one user population, and your own rate will differ, but nothing suggests it will be high enough to find failures with. Support tickets are worse as a signal, because a user who never trusted the assistant does not file a ticket about it.

When the agent is customer-facing and multi-tenant, the averaging is the whole problem. A broken index, an expired credential or a tool scoped to the wrong account fails completely for one tenant and perfectly for the other four hundred, and a global success rate moves by a fraction of a percent. Every detector below should be evaluated per tenant as well as globally.

How do I find bad AI responses users never complain about

You detect them from the shape of the conversation and the contents of the trace, so the first requirement is that the trace has enough in it. The minimum is a stable conversation identifier, per-turn timestamps, full assistant text, tool calls with their arguments and their results, and the tenant and end-user identity the turn ran under. Tool results are the field teams most often drop for size reasons, and dropping them removes half the detectors below. The field-by-field version of that list is in what to log from an LLM app, and the wider category is LLM observability.

The second requirement is one business-outcome table joined to the conversation. A refund, a credit, a cancellation or a discount code issued within a day of an assistant conversation is the strongest signal on this page, because it is the only one that carries a number in currency.

Signals worth detecting

Each of these is a deterministic query over stored traces. None needs a model, and all of them are cheap enough to run over every conversation rather than a sample.

Two rules keep these usable. Alert on rates against a rolling seven-day baseline rather than on single occurrences, and compute the rate per tenant, per surface and per agent version, since a signal that doubles for one tenant is invisible in the global number. Abandonment is also an adoption signal, so a rising rate is worth reading alongside the numbers in measuring AI feature adoption.

The empty-result detector, which is the one that catches the indexing failure described above, is a single query:

select c.conversation_id, c.tenant_id, c.started_at
from tool_calls t
join assistant_turns a
  on a.conversation_id = t.conversation_id
 and a.turn_index = t.turn_index
join conversations c
  on c.conversation_id = t.conversation_id
where t.tool_name = 'search_knowledge_base'
  and t.result_count = 0
  and length(a.content) > 200
  and a.content not ilike '%could not find%'
  and c.started_at >= now() - interval '7 days'
order by c.tenant_id;

Run the review loop weekly

1. Fix the population before you sample

Decide what a conversation is (a session, a thread, a run) and write the definition down, because the abandonment detector is meaningless if the boundary moves between weeks. Exclude your own staff traffic, load tests and health checks at query time. Keep the tenant identifier on every row so the sample can be balanced later.

2. Score every conversation with the cheap detectors

Run all eight queries over the full week and store the flags on the conversation, one boolean column per signal. Storing them is what makes the next week comparable and lets you measure each detector's precision after review. A conversation carrying three flags is worth more attention than one carrying a single flag, so keep the count.

3. Sample with quotas, and include a control

Take about seven conversations per detector rather than a proportional sample, since the rare signals are the informative ones. Add ten unflagged conversations as a control group: if the control turns out to contain failures at a similar rate, your detectors are not finding what you think they are. Weight the quota toward tenants where a failure costs the most, and record how the sample was drawn so the next week can repeat it.

4. Label against a three-way rubric

Read each conversation and assign one of three labels: fine, degraded (the user got there, with friction), or wrong (the answer was incorrect, or the task failed). Write one free-text sentence naming the cause, in the reviewer's own words, and resist the temptation to pick from a list on the first pass. Have a second reviewer independently label ten of the seventy and compare, because a rubric two people apply differently produces a ranking nobody should act on.

5. Cluster the causes and rank by cost

Group the free-text causes into named classes after the fact, which is how you learn that "empty retrieval answered anyway" and "stale index" are the same bug. Rank the classes by estimated frequency in the population times the cost of one instance, using the refund-shaped outcomes as the cost anchor where you have them. Three or four classes usually account for most of the weight, and the long tail can wait.

6. Turn the top classes into cases and standing alerts

Every class at the top of the ranking gets two artifacts: a regression case built from a real conversation, and a detector left running with a per-tenant threshold. The case is what stops the bug returning after the next prompt or model change, and the method for building one from a real trace is in building an eval set from production. Keep the review loop running afterward at a lower quota, since the classes you have named are the ones you have stopped discovering.

Where this gets easier

Runtype records each execution with per-step inputs and outputs, tool calls with their arguments and results, latency and cost, and its structured logs and record queries let you filter that history down to the conversations a detector flagged, per tenant and per surface. Any execution you surface can be promoted into an eval case, so a failure class you found by reading turns into a regression test in a suite instead of a note in a document. Judge scoring with human review of individual scores covers the labeling step at a larger sample size than a person can read, and coverage reporting shows which classes the suite still does not exercise.