Guardrails that still hold when the model changes

A guardrail survives a model change when the component enforcing it is something other than the model. Rules written into a system prompt are interpreted by whichever model is on the other side, so they change meaning when the model does. Controls enforced by code around the model (a schema the output must pass, a tool that is absent from the tool set, an approval the runtime demands, a counter that stops the fifth call) behave identically on a model you have never tested. The way to tell the two apart in your own system is a swap test run against adversarial cases before the new model reaches customers.

The failure is silent until someone screenshots it

Guardrail regressions do not raise errors. The agent responds, the response looks fine, and the one case where it quoted an internal margin figure or agreed to a refund it should have escalated sits in a log nobody reads. Teams find out when a customer posts a screenshot, which is weeks after the model changed.

The change is often one you did not schedule. Provider aliases move to new snapshots, a rate-limit fallback routes a fraction of traffic to a second provider, and a cost optimization sends short turns to a smaller model. Each of those is a different program running your prompt. Model upgrade regression testing covers the scheduling and rollout side of this; the question here is which of your controls need retesting at all.

Two other conditions move a prompt-layer guardrail without touching the model. Long conversations push the instruction far from the current turn, and the model weighs a 40,000-token tool result it just read against a sentence from forty turns ago. Unexpected phrasing routes around the specific wording of the rule, which is why the same restriction stated as a hypothetical, a translation exercise or a code comment gets past it.

What breaks each kind of guardrail

Sort your existing guardrails into this table before you plan a swap. The right column is the thing you have to test for; the guardrails whose breaking condition is "someone edits the config" need review, not re-testing.

Two rows deserve care because they look deterministic and are not fully so. A regex filter keeps working across models, and it also keeps missing everything phrased differently, so a model that gets more verbose can push output past it without any rule changing. An approval gate that matches on a tool name is deterministic only while the name matching holds; if the gate lives in application code that inspects a tool-call payload rather than in the runtime that dispatches the call, a change in tool-call formatting can route a call around it. LLM guardrails ranks the layers in more detail.

How do I stop my AI agent from saying or doing the wrong thing

Start from the consequence rather than the rule. For each thing you do not want to happen, ask three questions: can it be undone, does it cross a trust boundary, and would a customer see it. The answers pick the enforcement layer, and the rule is to use the strongest layer that can express the requirement, then add prompt wording on top as a first filter.

• An irreversible side effect (money moved, an email sent, a record deleted) belongs in tool scoping and approvals. Remove the tool from agents that do not need it, and gate the rest so a person sees the tool name and the exact parameters before it runs. Human-in-the-loop approval covers where the gate should live and who approves it.
• A decision your code acts on (approve or deny, an amount, a routing category) belongs in a schema. An enum of three values means the model cannot invent a fourth, and "maximum": 500 on an amount is a boundary no phrasing argues past. Decide the failure contract in advance: retry once with the validation error, then fail closed to the escalate branch, and never parse the prose instead.
• Data the caller must not see belongs at the retrieval boundary, scoped by the identity the request arrived under, before the model is involved. A prompt line saying "only answer about this customer's own orders" is a request; a query filtered by tenant is a fact.
• Runaway cost or looping belongs in counters. A tool that returns an empty array the model reads as "try again" produces the same retry storm on any model, and only a limit on calls per turn stops it.
• Anything reachable from text the agent reads belongs in the layers above, because the instruction channel is open to whoever wrote the document. Prompt injection when the agent has tools works through the mechanics.
• Tone, register and formatting stay in the prompt, and you accept a failure rate. Move them out only if a formatting failure has a consequence, in which case it belongs in the schema row instead.

Requirements that only the prompt can express (say, "do not speculate about a customer's medical condition") are the ones the swap test exists for. Write them down as claims you can test, not as prompt text you can only reread.

The model-swap test

The procedure is a before-and-after on the same adversarial suite, with the prompt held constant. Changing the prompt and the model together makes the result uninterpretable.

1. Write each guardrail as a testable claim. "The agent never sends an email to an address that did not appear in the request" is testable. "The agent behaves professionally" is not. One claim per line, with the enforcement layer from the table beside it.
2. Generate five to ten adversarial cases per claim. Vary the attack, not the wording: a direct request, the same request planted in a document or tool result the agent will read, the request buried at turn twenty of a long conversation, a hypothetical or role-play framing, and a paraphrase in another language. Cases that are trivially refused teach you nothing; aim for ones you are unsure about.
3. Define the assertion at the right layer. For a prompt-layer claim, assert on the response text. For a runtime-layer claim, assert on the trace: the tool call never reached dispatch, the run stopped at call ten, the approval was requested. A trace assertion catches the case where the model tried and the runtime stopped it, which is a pass, and the case where the gate silently did not fire, which is a failure the response text would hide.
4. Baseline on the model you run today. Record the pass rate per claim. A prompt-layer guardrail that already fails two of its ten cases on the model you trust is a filter you have been calling a boundary, and that is worth knowing before anything changes.
5. Run the identical suite on the candidate model. Compare case by case, not by aggregate score. Two suites can share a pass rate while failing on different cases, and the new failures are the finding.
6. Classify every regression by layer. A prompt-layer failure means rewriting the instruction or moving the requirement down a layer. A classifier failure usually means the threshold needs recalibrating against the new model's output distribution. A runtime-layer failure is a bug: those assertions should be model-independent, and when one fails the usual cause is a gate keyed on something the new model formats differently.
7. Promote every production failure into the suite. The screenshot a customer sent is the highest-value case you will ever get, and it belongs in the suite as a regression case before the fix is written, so the fix is verified rather than assumed.
8. Re-run on every model change, prompt change and tool addition. Adding a tool changes the action space, which can invalidate a claim that held when the tool did not exist.

Sampling matters when you read the results. Run each case more than once at your production temperature, because a claim that passes on one sample and fails on the next is a claim the model is deciding, and the correct response to that is to move it to a layer that does not sample.

Where this gets easier

Runtype enforces the durable layers outside the model, so a swap does not move them: an agent's tool set bounds what it can call at all, with secrets resolved server-side through {{secret:NAME}} references the model never sees; approval gates apply per tool or to every tool with a timeout, and the reason shown to an approver is the agent's own claim rather than an input to the decision; flow prompt steps can require JSON output and flows are validated at save time; and counters cap tool calls per turn (default 10, maximum 100), turns per run (1 to 100), tool duration (30 seconds by default, 60 for MCP) and the wall-clock budget for a turn. The swap test itself runs as an eval suite with cases promoted from recorded executions, judge scoring with human review of individual scores, and run-to-run comparison, so the before-and-after in step 5 is a stored comparison rather than a spreadsheet someone kept.