How to keep PII out of your LLM logs

Redact on the write path, before the record is persisted. Detect in the payload as it leaves the runtime, replace each match with a stable token, and keep the structural fields that debugging depends on. A masking rule applied when someone opens a trace has already failed, because the raw value reached storage, the search index and last night's backup.

The compromise that makes an agent undebuggable

Debugging an agent means reading what actually went in and out: the user turn, the system prompt after variable substitution, the tool arguments the model produced, the tool result it reacted to. Privacy review reads the same fields and sees a support ticket body, a shipping address, an account holder's name and a phone number, sitting in a third-party log vendor with a two-year retention default.

The usual settlement is a logging flag, turned off in production and on when someone is investigating. It produces a system where the only runs you can see are the ones you already knew to watch, and where the interesting failure happened three days ago in an account you cannot reproduce.

Turning capture off also removes the raw material for everything downstream. Production traces are where regression cases come from, and a team that logs nothing has no way to build an eval set from production when a model upgrade lands.

How do I stop customer PII ending up in my AI logs

Move the decision from the read path to the write path, then make it a policy rather than a per-call choice. Redaction belongs with the other checks that run on payloads entering and leaving a model, collected under LLM guardrails. The steps below run from what counts as PII in a conversational payload through detector selection, retention, and the tests that tell you the redaction still works after someone edits a prompt.

1. Write down what counts as PII in a conversational payload

Card numbers and email addresses are the easy tier, because they have a format and a checksum. What actually accumulates in agent logs is unstructured: a customer pasting an order confirmation, a name in a greeting, a home address dictated to a scheduling agent, a diagnosis mentioned in passing to a benefits assistant.

Separate the list into detectable formats and free-text categories, and be honest about which side each item falls on. Formats worth detecting by pattern include email addresses, phone numbers with a country prefix, credit cards validated by a Luhn check, IBANs validated by mod-97, national identifiers with a check digit, and IP addresses. Free-text categories, which no pattern will find, include personal names, street addresses, employer names, health details and anything a customer volunteers about a third party.

The second list matters more than the first for a customer-facing agent, because the field that fills up with it is the user turn itself, and that is the field you most want when reconstructing a failure.

2. Put the redaction between the runtime and the writer

The redaction call belongs in the code path that serialises a trace record, a log line or a conversation row, before it is handed to the store. Every writer has to go through it: the trace exporter, the structured logger, the conversation history table, the error reporter that attaches request context, and the queue message that carries a payload to a background worker.

One uninstrumented writer defeats the whole scheme. The usual survivor is an exception handler that ships the full request body to an error tracker, because that path was written before the agent existed and nobody thought of it as logging.

3. Replace matches with a stable token, not a fixed mask

Masking every match to the same string destroys the relation that makes a trace readable. If three tool calls in one run all show [REDACTED] in the email argument, you cannot tell whether the agent looked up one account three times or three accounts once, which is exactly the question a wrong-answer investigation turns on.

Emit a token that is deterministic per value and per tenant, derived from a keyed hash of the normalised value, and carry the entity type so the shape stays legible:

{
  "tool": "lookup_account",
  "arguments": {
    "email": "[EMAIL:a41f9c2e]",
    "order_id": "ORD-88213"
  },
  "result": {
    "account_id": "acct_5512",
    "billing_name": "[PERSON:7b30d1aa]",
    "shipping_line1": "[ADDRESS:e0c44f19]"
  }
}

Key the hash per tenant so the same email under two customers produces different tokens, which stops a token from becoming a cross-tenant join key. Keep the mask itself free of digits, at signs and IBAN-shaped prefixes, so running the detector over an already-redacted record finds nothing new and redaction stays idempotent.

4. Choose detectors with their failure modes in view

Three tiers exist, and most teams need two of them running together. Each fails in a different direction, which is the reason to combine rather than pick.

Run validated patterns unconditionally, because they are cheap enough to apply to every field and precise enough to trust. Add an entity model for the free-text fields where names and addresses land, and set its confidence threshold by measuring on your own transcripts rather than accepting the shipped default.

Treat a model-based pass as a reviewer rather than the gate. The text it inspects is attacker-controlled in a customer-facing product, so a message containing instructions to the redactor is a live path, and a redactor that fails open on a timeout is a redactor that stops working under load.

5. Treat tool results and retrieved documents as PII sources

The user turn gets the attention. The larger volume usually arrives from the other direction: a CRM tool returning a full contact record when the agent needed a status field, a retrieval step pulling three support threads into the context window, a database step returning every column of a row.

Redact tool results with the same policy as user input, and reduce what the tools return in the first place. A tool that projects four fields instead of a whole record shrinks the log surface, the context window and the blast radius of a prompt injection at once. Retrieved documents deserve their own rule, since a chunk retrieved for tenant A that mentions a person from tenant B is a tenancy failure before it is a logging one, which is why retrieval scoping sits on the tenant isolation checklist.

6. Keep the fields that make a redacted trace still worth reading

Redaction that removes structure produces records nobody opens. What survives should be everything that is not a value: tool names, argument keys, the JSON shape of results, array lengths, token counts split by cached and uncached, latencies, stop reasons, error types, model and version identifiers, retry attempts, and the tenant and end-user identifiers your own system issued.

Those identifiers are the exception worth arguing about. An internal account id is personal data under most regimes, but it is also the only way to answer "show me this customer's failed runs", so treat it as a field to scope and expire rather than one to hash away. Which fields carry the debugging signal, and which are volume without value, is worked through in what to log for LLM calls.

7. Make the policy a setting, per environment and per product

A single global switch forces one answer onto surfaces with different obligations. A staging environment with synthetic data wants full payloads; a healthcare tenant wants redaction on every field including tool results; an internal admin agent used by your own staff sits somewhere between.

Resolve the effective policy from a layered hierarchy, so a specific setting beats a general one and an unset value inherits:

product:
  pii_redaction: redact
  logging: on
surfaces:
  web_chat: { pii_redaction: default }
  internal_ops: { pii_redaction: off }
agents:
  billing_assistant: { pii_redaction: redact }

For nested execution, treat the policies as floors rather than ranked layers. A subagent or child flow invoked by a redacting parent should redact even if its own setting says otherwise, because the alternative lets a delegated call launder the parent's payload into an unredacted log.

8. Tier retention instead of choosing one number

One retention setting applied to everything is what makes privacy review ask for the logging to be turned off. Split the record instead: metrics and identifiers live long, redacted payloads live medium, and any unredacted capture lives days.

Wire deletion to the same tenant identifier the traces carry, so an erasure request resolves to a query rather than a search across log vendors. Any export that copies traces into a warehouse or an eval set inherits the tier it came from, and eval cases promoted from production keep the tokens rather than the raw values.

9. Test the redaction like any other guardrail

Redaction rots quietly. Someone adds a tool whose result includes a new field, a prompt starts echoing user input into a system message, a library upgrade changes a serialiser, and nothing fails.

Keep three tests running. A fixture suite feeds known values of each entity type through the real write path and asserts the stored record contains the token and not the value. A canary job writes a synthetic record with planted markers on a schedule and greps the store for those markers, which catches the writer nobody instrumented. A sampled scan runs the detector over already-stored records and alerts on any hit, since a hit means something reached storage unredacted.

Add idempotence and a false-negative measure to the fixture suite. Re-running the detector over its own output should find zero matches, and a labelled set of real transcripts, kept small and reviewed by hand, tells you what the entity model is missing before a customer does.

Where this gets easier

Every step above is code you own, applied consistently across writers that were built at different times by different people. Runtype treats PII redaction and logging verbosity as policy set per product, surface or agent and resolved at dispatch, with the stricter setting winning for nested runs, so a subagent cannot log what its parent redacted. Detection covers validated formats including cards checked with Luhn and IBANs checked with mod-97, redaction is idempotent by construction, and the traces themselves keep per-step inputs and outputs, tool calls with arguments and results, latency and cost, which leaves the debugging structure intact after the values are gone. The privacy answer becomes a setting on the product rather than a decision to stop logging.