What to log for an LLM feature, and what you will regret not logging

Capture four groups of fields per model call: what ran (prompt version, config hash, model requested and served, tenant, end user), what happened (each attempt, tool call arguments and results, latency), what it cost (input, cached input and output tokens against a pinned price table), and what the user did next. The last group is the one teams add after their first bad week.

The fields you have and the questions you get asked

A customer reports that the assistant quoted a refund window of 14 days when their plan gives 30. You open the log and find a prompt and a completion. The prompt is the rendered text, which is better than most teams have, and it is still not enough to answer any of the questions that follow.

Which version of the system prompt produced it, given that the template shipped twice on Tuesday. Which model served the call, given that the code asks for an alias. What the order lookup tool returned, given that the log records only that a tool was called. Whether this user tried three more times and then emailed support, or accepted the answer and closed the tab.

Each of those is a field somebody decided not to write, usually because the log was designed while a single engineer watched calls scroll past in a terminal. The per-call record underneath this list, and why aggregate dashboards miss the failures that matter, is covered at LLM observability.

What should I log for my LLM application

Group the capture list by the question each field answers after the fact. Fields that answer no question you will actually be asked are cost and privacy risk with no return.

1. Pin what ran

Record a prompt version identifier (a template id plus a monotonic version, not a git SHA of the whole repo) and a config hash computed over the model id, sampling parameters, the system prompt hash and the sorted tool names with their JSON schema hashes. The config hash catches the change nobody thinks of as a prompt change: an engineer adds an optional field to a tool schema, model behaviour shifts, and the prompt version is untouched.

Record the model requested and the model served as two fields. Aliases such as a -latest suffix resolve to a dated snapshot on the provider side, and a snapshot rollover looks exactly like a random quality regression until the two fields disagree in a query.

Record the provider request id from the response (id in the response body, or the x-request-id header the provider echoes). It is the only field in the record you cannot reconstruct later, and it is the first thing a provider support ticket asks for. Record the tenant and end-user identity in the same layer that makes the call, not in each caller, since the background job and the webhook replay are the callers that forget.

2. Record every attempt, not every call

Write one entry per attempt, each with an attempt number, a retryOf pointer and the failure class of the attempt it replaced (timeout, 429, 5xx, schema validation failure). The Anthropic and OpenAI clients for Python and TypeScript all default max_retries (maxRetries in TypeScript) to 2, so one call can be three billed attempts and nine seconds that reach your log as a single entry. Either set that option to 0 and run the retry loop yourself, or observe each attempt from a layer the client hands every request to: a custom http_client in the Python SDKs, the middleware array in the Anthropic TypeScript SDK, a wrapped fetch in the OpenAI Node SDK.

For each tool call, record the name, the arguments exactly as the model emitted them, the result or the error, the duration, and whether the result was truncated before it went back to the model. Truncation is the field most stacks skip: a lookup returns 40 KB, a helper cuts it to fit the context window, and the model answers from the half it saw. Nothing else in the record explains that answer. How to nest these entries into a tree with one root per user interaction is covered at agent tracing.

Record what happened to the output after the call returned: whether it parsed, which validator rejected it and with what message, and whether a fallback value was substituted. A structured-output feature that silently falls back is the most common shape of a silent failure in an AI feature.

3. Record tokens the way the invoice is computed

Store input, cached input, output and reasoning tokens as separate counts, taken from the provider response rather than from a local tokenizer. Cached input is billed at a fraction of the uncached rate, so a record with one input number cannot be reconciled against a bill, and a cache that quietly stopped hitting looks like a price increase.

Store the computed cost as a number alongside the identifier of the price table you priced it with, such as a date. Repricing a table without that identifier rewrites the history of every past record, which is how a cost investigation ends in an argument about whether last quarter really was cheaper.

4. Record what the user did next

Append an outcome row keyed on the generation id rather than mutating the original record, because outcomes arrive seconds to days later and the generation record is worth more immutable. Useful outcomes are narrow and mechanical: the user sent another message within 60 seconds whose embedding is close to the previous one (a rephrase), the user thumbed down, a human took over the conversation, a ticket was created, the caller retried with different parameters, or the answer was copied or accepted into a form.

These four or five booleans do more for quality work than any offline metric, because they are the only fields in the record that know whether the answer worked. They are also the raw material for building an eval set from production traffic, since a rephrase-and-escalate pair is a labelled failure that cost you nothing to collect.

5. Set retention and redaction per field, not per log

Split the record into a payload tier (rendered prompt, completion, tool arguments, tool results) and a skinny tier (ids, versions, token counts, cost, timings, outcomes). Give the payload tier a short window with a hold flag that exempts anything under investigation, and give the skinny tier a long one, since it is small and carries little personal data.

Apply redaction when the record is written, not when it is read, and apply it to tool arguments and tool results as well as to the prompt. An order lookup result carries the same personal data as the message that triggered it. Store the redaction policy version on the record so a later reader knows what a blank field means, and keep a hash of the unredacted prompt so two runs can still be proven identical after the payload has expired.

Replacing an identifier with a hash salted per tenant preserves the ability to count how many sessions saw the same email address without storing one. The full treatment, including what to do about data that arrives inside a retrieval result, is at PII redaction in LLM logs.

Field-by-field capture list

Retention values below are starting points to argue about with whoever owns your data policy, not requirements.

A sample record

The generation record, written when the call returns:

{
  "generationId": "gen_01J9Y7C4",
  "runId": "run_01J9Y7BX",
  "attempt": 2,
  "retryOf": "gen_01J9Y7C1",
  "prevAttemptError": "429 rate_limit",
  "tenantId": "acme",
  "endUserId": "eu_8812",
  "surface": "web-chat",
  "promptVersion": "refund_policy@11",
  "configHash": "cfg_5c1e0a3f",
  "provider": "openai",
  "model": { "requested": "gpt-4o-mini", "served": "gpt-4o-mini-2024-07-18" },
  "providerRequestId": "req_9d41c07b",
  "prompt": { "hash": "sha256:41b9c2", "redacted": true, "policyVersion": "pii-v3" },
  "response": {
    "stopReason": "length",
    "parsed": false,
    "validationError": "refund_amount: required"
  },
  "toolCalls": [
    {
      "name": "lookup_order",
      "arguments": { "orderId": "AC-44810" },
      "status": "ok",
      "resultTruncated": true,
      "durationMs": 310
    }
  ],
  "tokens": { "input": 3120, "cachedInput": 2560, "output": 800, "reasoning": 0 },
  "timingMs": { "firstToken": 710, "total": 4180 },
  "cost": { "amount": 0.0042, "currency": "USD", "priceTable": "2026-08-01" },
  "retention": { "payloadUntil": "2026-10-03", "recordUntil": "2027-10-03", "hold": false }
}

The outcome row, appended later against the same id:

{
  "generationId": "gen_01J9Y7C4",
  "observedAt": "2026-09-03T14:22:07Z",
  "kind": "user_rephrase",
  "detail": { "secondsSince": 41, "similarity": 0.91, "escalatedToHuman": true }
}

Three fields carry most of the diagnosis here. A stopReason of length beside parsed: false says the completion hit the token cap mid-object and a fallback went out, without anyone reading the text. The resultTruncated flag says the model never saw the shipping date it was asked about. And the outcome row, arriving 41 seconds later, says the user did not accept the answer, which is the fact that decides whether this record becomes an eval case.

Where this gets easier

Runtype records this set as a property of running the call rather than as instrumentation somebody remembered to add: execution traces carry per-step input and output with latency, every tool call with the arguments the model produced and the result it received, and cost per execution, per record and per batch with cached and uncached tokens reported separately. PII redaction and logging verbosity are policies set per product, surface or agent and resolved at dispatch, so a customer with a stricter data agreement changes a policy rather than making you turn logging off for everyone. Structured logs with aggregate log stats sit beside the traces, and an execution can be promoted into an eval case directly, which is the outcome-field loop described above without a second pipeline. For a feature already built elsewhere, the same records can be populated from OpenTelemetry traces sent to https://api.runtype.com/v1/otel.