A QA process for AI features that a team without ML engineers can run

QA for an AI feature works once the release gate is written down before the feature is built. Four stages carry it: an exploratory pass that names the failure classes, a frozen regression set that has to stay green, a weekly review of sampled production output against a written rubric, and one live metric with a threshold that triggers rollback.

None of that requires an ML engineer. It requires a rubric a support lead can apply in about two minutes per case, roughly forty saved cases, and one named person who decides whether the release goes out.

The problem your existing test plan runs into

A test plan has an expected-result column, and that column is where the process breaks. Run the same input twice and the wording changes, so a tester files a defect and an engineer closes it as working as intended. After that happens twice, testing quietly becomes a demo, and the shipping decision moves to whoever sounds most confident in the room.

The failures that matter are rarely crashes. Silent failures arrive as completed executions: a search tool returns an empty array, the model reads that as "there are no results", and the answer is a fluent, confident, wrong paragraph delivered with a 200 status code. Deterministic tests miss it because nothing threw.

Blocking forever is the opposite failure. With no agreed definition of good enough, every reviewer applies a private bar, and a feature sits in review while its own team argues about tone.

How does QA work for an AI feature

It runs as four stages with different owners, different cadences and different artifacts. Two happen before a release, one repeats every week after it, and one is a number you agree on in advance and then instrument. The set of stages is the same whether the feature ships to one internal team or to every customer on your platform, though the sampling and the thresholds change when it is customer-facing.

1. Exploratory pass: name the failure classes

Two people who did not write the prompt each spend a day sending real inputs at the feature. Draw the inputs from support tickets, search logs and the queries your sales engineers get asked in demos, not from imagination. The output is a list of failure classes, each with a one-line description and one input that reproduces it, rather than a count of passes and failures.

Classes that recur: refusal on a question that is in scope, a confident wrong fact, an answer correct in content but citing another customer's data, a tool called with an argument the API rejects, truncation mid-sentence when the token cap is hit, and format drift where a request for JSON returns JSON wrapped in a code fence with a sentence above it.

Stop when a full session produces no new class. That usually takes two days of two people, and the list is the input to every other stage.

2. Freeze a regression set

Turn each failure class into cases, and build the rest from real traffic. Building an eval set from production is faster and more honest than writing cases by hand, because production already contains the inputs your team would never have thought of. A case is an input plus assertions, never an expected string:

- id: refund-outside-window
  input: 'I bought this 63 days ago and I want a refund.'
  assertions:
    - type: must_not_contain
      value: 'refund has been issued'
    - type: json_schema
      require: ['decision', 'policy_reference']
    - type: judge
      rubric: policy-faithfulness
      min_score: 3
  source: ticket-48120
  failure_class: confident-wrong-policy

Three assertion types cover almost everything: structural (the JSON parses and carries the required fields), forbidden content (never states a refund was issued, never names another tenant), and judged (a rubric score at or above a floor). Judged assertions need calibration before you trust them, so read how LLM-as-a-judge scoring works and check the judge against a human on twenty cases. If the two disagree on more than one in five, fix the rubric before the score gates anything.

The set is frozen for a release. Cases get added freely, and an existing assertion is edited only with a note saying who changed the bar and why.

3. Review sampled production output weekly

Pick fifty executions a week and stratify them, because a random sample of live traffic is mostly easy questions. A workable split is twenty random, fifteen drawn from the longest or most expensive turns, and fifteen that hit a tool error or a fallback path. One reviewer, one sitting, a written rubric with four verdicts.

Use a sheet with fixed columns so the week-over-week counts are comparable:

The promotion rule is mechanical, which is what keeps the set from bloating. Every unacceptable and every harmful verdict becomes a regression case before the next release, using the reviewer's note as the assertion. A minor verdict is counted and not promoted, unless the same failure class appears three times in a month, at which point it is promoted too.

For a multi-tenant feature, sample per tenant rather than across all traffic. One large account can supply eighty percent of executions, and an aggregate that looks healthy can hide a smaller tenant whose every answer is wrong because their data model differs.

4. Agree one rollback metric and its threshold

Pick a single live metric, write its threshold and window before launch, and instrument it. Candidates that work: the unacceptable rate in the weekly sample, the rate at which conversations escalate to a human, the share of turns ending in a tool error, or negative feedback per hundred conversations. Whichever you choose, the sentence has to be specific enough to act on without a meeting, for example: escalation rate above twelve percent of conversations over any rolling twenty-four hours rolls the feature back to the previously published version.

Three dashboards and no threshold is the common failure here. So is a threshold measured only in aggregate, which for a customer-facing feature will not fire when one tenant is fully broken and the rest are fine.

Who owns what

Rollback authority sits with the on-call engineer and needs no approval, which is what makes the threshold worth writing.

The release checklist

A release goes out when all six lines are true and someone has said so in writing:

• The regression set is green on the candidate version, and the run is linked from the release ticket.
• Every failure class from the exploratory pass is either fixed or accepted in writing, with a named owner and a date.
• Judged assertions were calibrated against a human within the last thirty days.
• The rollback metric, threshold and window are written down and firing on real data.
• A named reviewer owns the weekly sample for the first four weeks.
• The previous version is still deployable, and someone has confirmed that this week.

Keeping the regression set green through prompt edits is its own discipline, covered in regression testing prompt changes. The wider practice these four stages belong to, including how suites, cases and coverage fit together, is AI agent evals.

Where this gets easier

The gate only holds when it runs by itself, and most of the friction above is bookkeeping: which cases exist, which version they ran against, and whether last week's reviewer verdict ever became a case. Runtype keeps eval suites and their cases attached to the flow or agent they test, promotes a recorded execution into a case in one step, records judge scores with human review of individual scores, and reports coverage so an untested failure class is visible rather than assumed. Publishing a version is not itself blocked by a red suite: the enforcement point is your pipeline, where runtype eval run exits non-zero when a gate grader fails and --strict extends that to soft graders (managing evals as code). Run-to-run and record-level comparison turns "must stay green" into a diff you read before publishing, and execution traces carrying per-step input and output, tool calls with arguments and results, and cost per execution give the weekly sample and the regression cases a single source (what evals are).