Why your agent gives a different answer every time, and what to do about it

Some of the variance is yours and some of it belongs to the provider. Sampling settings, retrieval order, tool results and how you assemble the conversation are under your control. Batched inference on shared hardware and provider-side model updates are not. Split the two, remove the first, then measure what remains as a rate over many runs rather than comparing two answers.

The rest of this page names each source and what it costs to pin down, corrects two beliefs about temperature 0 and seeds, then gives the steps for measuring and bounding what is left.

My AI gives different answers to the same question

A customer screenshots two contradictory answers from the same feature. Support pastes the question into staging, gets a third answer matching neither, and closes the ticket as not reproducible. QA refuses to sign off because the acceptance criterion was written as an expected string and the string moves.

When the agent is customer-facing, the reproduction gap is wider than the model. The customer's tenant has different records, a different tool configuration and a different conversation history, so the question alone was never the input. Reproducing from a screenshot means reconstructing the whole request, which most teams discover they never recorded.

Where the variance comes from

Six sources, in rough order of how often they turn out to be the actual cause once a team starts measuring.

Sampling

Temperature divides the logits before the softmax, and top-p or top-k truncate the candidate set. Higher temperature widens the distribution the decoder draws from, so two runs pick different tokens early and diverge thereafter. This is the source everyone reaches for first, and at a default temperature around 0.7 it is genuinely the largest one.

Batched inference and floating-point non-associativity

Your request is batched on the server with whatever traffic arrived at that moment, and the batch composition follows load you cannot see. Floating-point addition is not associative, so a reduction over thousands of terms gives a different last-bit result when the kernel splits the work differently, which it does when the batch shape changes. The effect on any single logit is tiny. It matters only when two candidate tokens are nearly tied, and then it flips the choice and the answer follows a different branch.

The common explanation, that GPU atomics introduce concurrency-dependent ordering, matters less than kernels whose reduction strategy is not invariant to batch size. Thinking Machines Lab's September 2025 post, Defeating Nondeterminism in LLM Inference, showed that one matrix multiplication repeated on a single GPU is bit-identical, and traced endpoint variance to batch size moving with load.

Mixture-of-experts routing can add a second batch dependence, because an implementation that enforces a per-expert capacity limit drops the tokens overflowing it, and which ones overflow depends on the others sharing the batch. Hosted providers do not say whether their models do this, so treat it as a candidate rather than a confirmed cause.

Provider-side model updates

An alias like a family name resolves to whichever snapshot the provider currently serves, and that mapping changes without a deploy on your side. Pinning a dated snapshot identifier removes most of it. Pinning is a lease rather than a guarantee: snapshots are retired on a published schedule, and the serving stack, safety filtering and default behaviour around a pinned snapshot can still change. Planning for the retirement is covered in surviving a model upgrade.

Retrieval

Vector search over an approximate index is approximate by construction. Graph-based indexes traverse a neighbourhood that shifts as documents are inserted or deleted, and equal-scoring results are tie-broken by an internal document id that changes when the index is rebuilt. Re-embedding the corpus with a new model version changes the scores outright. Any of these reorders the context, and reordered context is a different input.

Tool results

A tool that reads live data returns different data. Less obvious: a tool reading stable data can still return it in an unstable order, because a query without an ORDER BY has no ordering contract and a paginated API may not either. When the model issues several tool calls in one turn and your runtime appends results in completion order, network timing decides what it reads next.

Context assembly and history

History truncated at a token budget moves its cut point as earlier turns grow. A summarisation step compressing old turns is itself a model call with its own variance, feeding the next one. A timestamp interpolated into the system prompt changes the prefix on every request, which also defeats prompt caching. Each is an input change that reads to the team as a model change.

What temperature 0 and a seed actually guarantee

Temperature 0 switches the decoder from sampling to taking the highest-scoring token. It removes exactly one source from the list above. It does not make the scores identical between requests, so the near-tie flip described earlier still happens, and low temperature has nothing to do with correctness: a confidently wrong answer becomes a reliably wrong answer.

Seed support is uneven as of September 2026. OpenAI's Chat Completions takes seed and returns system_fingerprint, documented as a Beta best effort with determinism not guaranteed, and its newer Responses API has no seed parameter. Gemini takes seed in generationConfig on the same best-effort terms, with no fingerprint. The Anthropic Messages API has neither.

When the fingerprint changes, repeatability ends and nothing in your code changed. Real bit-level determinism is available if you serve the model yourself with a fixed batch and batch-invariant kernels, which is a serving project, not a flag.

The practical consequence is that an assertion of the form output == expected_string is not a test. It is a coin flip whose bias you have not measured.

Measure and bound the variance in seven steps

1. Record the resolved request, not the prompt

Store what went to the provider on every run: the message array after template substitution, the model identifier including its snapshot, temperature and top-p, the tool definitions offered, and every tool call with its arguments and response. A run id support can quote turns "cannot reproduce" into a replay. Without it, every step below is guesswork.

2. Replay with the inputs frozen

Run one reported case twice: once with recorded tool responses and retrieval results replayed as fixtures, once live. If the frozen replay is stable and the live one is not, the variance is in your inputs and you can remove it. If it still varies, the cause is model-side and you move to bounding rather than removing.

3. Measure a rate, not a diff

Run the same frozen input 30 times and count how often each property you care about holds. Report it as an interval, because 29 of 30 is not 97 percent:

import math

def wilson(successes, n, z=1.96):
    p = successes / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return center - half, center + half

print(wilson(29, 30))  # about (0.833, 0.994)

Thirty runs separates reliable from flaky. Separating 95 percent from 99 percent takes several hundred, so spend that only where the difference changes what you ship.

4. Write assertions on properties, with a rate attached

A case is a set of properties and the rate each one must hold at. Hard requirements get a rate of 1.0 and any failure is a red run; softer ones get a threshold:

{
  "case_id": "refund-requested-past-window",
  "runs": 30,
  "assertions": [
    { "id": "calls_get_order", "type": "tool_called", "tool": "get_order", "min_rate": 1.0 },
    { "id": "states_window", "type": "field_equals", "path": "policy.window_days", "value": 30, "min_rate": 1.0 },
    { "id": "never_promises_refund", "type": "absent_claim", "claim": "refund approved", "max_rate": 0.0 },
    { "id": "tone", "type": "judge", "rubric": "polite-and-specific", "min_mean_score": 0.8 }
  ]
}

The categories that survive rewording are the same four in most suites: a tool was or was not called, a structured field has a given value, a required fact is present, a forbidden claim is absent.

5. Remove the variance you can remove

Pin the model snapshot rather than the alias. Drop temperature for extraction and classification steps, and leave it up where the job is drafting prose. Freeze the retrieval index for the test suite and sort results by a stable key. Sort tool results before they enter the message array, and truncate history at a turn boundary rather than a token count.

6. Constrain the shape, then move the facts out of the model

A schema-constrained response removes variance in structure even where wording still moves, which makes downstream code deterministic even when the generation is not. The mechanics of getting a model to hold a shape are in structured output from an LLM. The second half is deciding which parts of the answer the model is allowed to author at all: a price, a policy window or an order status should arrive from a tool and be rendered, with the model composing around those values rather than recalling them.

7. Gate on the distribution in CI

Run the suite on every prompt, model or tool change and compare rates against the recorded baseline for the previous version, rather than a fixed number invented at design time. A drop from 1.0 to 0.9 on a hard assertion blocks the merge; a two-point move on a judge score is something to read, not a blocker. How to build and maintain that suite is in testing a prompt change before it ships, and the wider practice sits under AI agent evals.

Design for the variance you keep

Put the non-deterministic step inside deterministic control flow. A fixed pipeline that calls the model for one classification and then branches in code has a variance surface of one step, while an open agent loop that decides its own path has one that grows with every turn. The trade-off between the two shapes is worked through in agents versus workflows.

Two habits make the remaining variance survivable in a customer-facing product. Make actions idempotent, so a retried turn phrased differently does not create a second refund or ticket. Surface a run identifier in the support view, so a screenshot resolves to an exact execution with its exact inputs instead of an argument about what the user typed.

Then set an explicit budget. Some behaviours must hold every time, such as never quoting a price the tool did not return. Others are allowed to vary, such as the ordering of three suggestions or an explanation's length. Writing that line down lets QA sign off on a system that will never produce the same bytes twice.

Where this gets easier

Most of step 1 is bookkeeping nobody wants to own. Runtype records the resolved input, the config version and every tool call with its arguments and results for each execution, and keeps cost and latency beside them, so a support ticket resolves to a specific run and a diff against the run that behaved. Eval suites replay cases against a recorded baseline, with judge scores reviewable case by case, turning "it changed" into "this assertion moved from 1.0 to 0.9 between these two published versions".