How to tell if a prompt change broke something before your users do

Treat the prompt as a deployable artifact. Version it, pin a named set of 30 to 50 recorded cases to that version, and run the set on every edit before it merges. Pass criteria assert on facts, tool calls and refusals rather than exact wording, so a rephrase stays green and a behaviour change goes red.

The rest of this page is the mechanics: where the cases come from, what a criterion asserts, which grader scores it, what threshold blocks a merge, and what to do when the diff is red on purpose.

The Friday edit nobody can review

Someone tweaks a system prompt on Friday to fix one rambling answer by adding "be concise and do not repeat yourself". That edit also removes the sentence that used to tell the user how to reach support, and the pricing question that used to call get_plan now gets answered from whatever the model remembers about your docs. Three behaviours changed. A reviewer looking at the pull request saw seven added words and approved them, because a prompt edit produces no diff anyone can reason about and no test that goes red.

Blast radius is wider when the agent is customer-facing and one prompt serves every tenant. A regression that only appears for the customers with a 4,000-item catalogue, or a strict no-refund policy, is invisible in the two conversations you spot-checked from your own demo account.

How do I know if changing my prompt broke my AI feature

Run the same fixed set of inputs through the old prompt and the new one, then compare criterion by criterion rather than reading the text. The comparison is only trustworthy if the inputs are frozen, the tool responses are frozen, and the criteria describe behaviour instead of phrasing. Building that is six steps.

Build the suite in six steps

1. Give the prompt a version identifier

Move the prompt out of the code path into a file with a version in its name, such as prompts/support-agent/v14.md, read at boot from a lockfile or config entry. Every recorded execution stores which version produced it. That single change buys a reviewable diff in the pull request and an eval run that can say which prompt it scored.

2. Draw the cases from real conversations

Synthetic cases test the behaviour you imagined. Production traffic tests the behaviour you have. A workable mix for a 40-case suite:

• 16 sampled from the last 30 days of real conversations, stratified across tenants so the large and unusual accounts are represented, not just the median one.
• 10 promoted from past incidents. Every bug someone reported becomes a permanent case the day it is fixed.
• 8 adversarial: prompt injection attempts, out-of-scope requests, users who insist the agent is wrong, empty or single-word inputs.
• 6 covering rare paths that production has not produced yet, such as a tool returning an empty array or an error.

De-identify each case as you capture it, and record the tool calls the original execution made along with the responses they returned. Those recorded responses become fixtures. Building the capture pipeline is its own job, covered in building an eval set from production traffic.

3. Write criteria that survive rewording

A criterion asserts something the model must do, not something it must say. Four families cover most cases: a tool was or was not called, a structured field has a given value, a required fact is present, a forbidden claim is absent.

{
  "id": "refund-request-past-window",
  "input": "I bought this 45 days ago and I want my money back.",
  "tenant": "acme",
  "tool_fixtures": {
    "get_order": { "purchase_days_ago": 45, "plan": "standard", "status": "delivered" }
  },
  "criteria": [
    { "type": "tool_called", "tool": "get_order" },
    { "type": "tool_not_called", "tool": "issue_refund" },
    { "type": "fact_present", "fact": "the 30 day return window has closed" },
    { "type": "pattern_absent", "pattern": "(?i)\\b(full refund|refunded in full)\\b" },
    { "type": "judge", "rubric": "Offers store credit or an exception path. Does not promise money back.", "min_score": 3 }
  ]
}

Exact-string comparison belongs only where the output is structured: a JSON field, an enum value, a routing label. Anywhere the model writes prose, an exact-match assertion goes red on every harmless rewrite and teaches the team to ignore the suite.

4. Choose one grader per criterion

Deterministic graders carry the load. A suite where most criteria are code and a minority are judged runs in under a minute and costs almost nothing, which is what keeps it in the merge path. Judges cover what nothing else can express, and their prompts get versioned and regression-tested like any other prompt, as set out in using an LLM as a judge.

5. Set thresholds that actually block

• Hard criteria (tool calls, schema, forbidden patterns) pass at 100%. One failure blocks the merge. These are the criteria a human wrote down as a rule, so a single violation is a defect.
• Judged criteria are compared against the baseline run, never an absolute bar. Block when the mean score falls more than 0.2 below the baseline mean, or when any single case drops more than 1.0 point.
• Run each judged case three times and take the median. Two of three runs disagreeing on a case with no prompt change means the case is unstable, not that the prompt regressed.
• A case that flips without a prompt change twice in a month gets quarantined and rewritten. A tolerated flake is a criterion the team has already stopped reading.

6. Gate it in CI

The gate runs on any pull request that touches the prompt directory, the tool definitions or the model configuration, and it compares against the currently deployed version rather than the previous commit.

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'tools/**'
      - 'config/models.yaml'
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Prompt regression suite
        run: |
          ./bin/evalrun \
            --suite support-agent-regression \
            --candidate prompts/support-agent/v15.md \
            --baseline  prompts/support-agent/v14.md \
            --fixtures  fixtures/support-agent \
            --repeat 3 \
            --fail-on hard-criteria,score-drop \
            --report out/diff.json

Add a nightly run of the same suite against live tools. The gate answers "did my prompt change break this", and the nightly run answers "did something under the prompt change", which is a different failure with the same symptom. Both belong in the wider practice of evaluating AI agents.

Read the diff, not the outputs

A useful report is a table with one row per criterion that changed, not a wall of before-and-after text. Each row carries the case id, the criterion, the old result, the new result and the direction it moved. Three shapes show up:

1. A hard criterion flipped from pass to fail. The prompt change caused it, because the input and the tool responses were identical. This blocks.
2. A judged score moved inside the threshold. Nothing blocks, and the row is still worth reading, because a slow drift across many cases is what a series of small edits looks like on the way to a rewrite nobody approved.
3. A case flipped from fail to pass. Either the edit fixed it, which is the point, or the criterion was wrong and the new behaviour happens to satisfy it. Check which before celebrating.

What to do on a red run

Reproduce the single red case against the old prompt first, with the same fixtures. If it passes there and fails on the new prompt, the edit is responsible and the classification is regression, intended change or bad criterion.

For a regression, narrow the edit rather than reverting it. A prompt instruction that fixes one case and breaks four is scoped wrong: it belongs in a tool description, a per-tenant configuration value or a conditional branch, not in the shared system prompt every tenant and every request path reads. Adding "be concise" to the top-level prompt is how the support-contact sentence disappeared.

Never change a criterion and the prompt in the same commit. Splitting them into two reviewed changes is the only thing standing between a regression suite and a suite that has been quietly edited until it agrees with whatever the model now does.

Re-blessing a baseline when the change is correct

Sometimes the new behaviour is right and the recorded expectation is stale, because the policy changed or the earlier answer was wrong. Update the case in its own pull request, one criterion at a time, with the reason written in the commit message and a second reviewer who did not write the prompt edit.

Two rules keep this honest. Re-bless individual criteria, never a whole run, since an "accept all" button turns the suite into a recording of current behaviour. Keep the superseded expectation in the case file's history so anyone can see what the agent used to do and when that stopped being required. A model version bump produces the same situation at much larger scale, which is why it gets its own procedure in surviving a model upgrade.

Where this gets easier

Runtype versions every agent and prompt configuration with draft and published versions. A suite attaches to a flow or agent rather than to a pinned version, so the comparison it gives you is this run against the previous run of the same cases: run it before the edit, run it after, and read the two. Cases can be promoted straight from a recorded execution, judge scores are reviewable one score at a time by a human, coverage reporting shows which behaviours no case exercises, and run-to-run and record-level comparison are part of the suite model instead of a diff script somebody maintains. The eval model is documented at what are evals.