What to do when the conversation outgrows the context window

Budget the window instead of appending to it. Decide up front how many tokens each part of the prompt gets, system instructions, tool definitions, retrieved material, conversation history and output headroom, then enforce those limits on every turn. Trim tool results, summarize old turns with a summarizer you have tested, and pin the constraints that must survive.

My AI agent forgets things in long conversations

The report is usually the same shape. Somewhere past turn twenty-five the agent contradicts an answer it gave earlier, drops a constraint the user set in the first message, or starts returning a provider error about token limits in the middle of a tool call. Nothing changed in the prompt or the model version. What changed is that the assembled prompt crossed a size where the parts you care about stopped being read reliably.

Three separate faults hide under that one symptom. The transcript plus tool results outgrew the window and your framework quietly dropped something. Everything still fits, but the early constraint now sits in the middle of a long context where recall is weakest. Or the provider rejected the request outright because an assistant message carrying a tool call lost its matching result during eviction.

Each has a different fix, which is why measurement comes before a bigger model.

A worked token budget

Give every part of the prompt a number and a rule for what happens when it overflows. The numbers below are illustrative rather than measurements from a real workload: replace each one with a count from your own tokenizer on your own traffic. The shape is what transfers.

The window here is 200,000 tokens and the working ceiling is 90,000. That gap is deliberate. Prompt cost, time to first token and the odds of a mid-conversation eviction all grow with the assembled size, so the ceiling is where you decide the conversation must change shape rather than where the provider decides for you.

Two things there matter more than the arithmetic. The first three rows are identical on every turn, which makes them the part worth prompt caching, and the reserved output block stays reserved. A run that borrows output headroom to fit two more turns of history produces a truncated answer at exactly the point where the conversation is most complex.

Retrieved material and tool results are what usually break the budget. A search tool that returns forty rows of full objects at roughly 300 tokens each costs 12,000 tokens per call, so an agent that calls it three times in one turn has spent the entire transcript slot on data it will compress away ten turns later.

Six steps that keep a long conversation inside the window

1. Measure what fills the window

Log the split, not the total. Count tokens per slot immediately before each model call and emit them as structured fields alongside the turn number, so a long conversation produces a series you can graph rather than a single number you can only compare to the limit.

{
  "turn": 31,
  "tokens": {
    "system": 1487,
    "tool_definitions": 6412,
    "pinned": 780,
    "retrieved": 11930,
    "summary": 1902,
    "transcript": 71204,
    "output_reserved": 640
  },
  "assembled_total": 94355,
  "ceiling": 90000,
  "window": 200000
}

That line is the diagnosis for one turn. The transcript slot is 16,204 tokens over its budget and the request is 4,355 over the ceiling, which says the eviction rule never ran. Without the split you would have concluded the model is forgetful.

2. Trim tool results before they reach the transcript

Cap results at the tool boundary, where the shape of the payload is still known. Three rules cover most of it: return only the fields the agent uses, cap each result at a token count, and keep the full payload behind an identifier the agent can fetch when it genuinely needs the rest.

A sketch of the shape, whatever your framework calls these fields:

tools:
  - name: search_tickets
    result:
      max_tokens: 1200
      fields: [id, subject, status, updated_at]
      overflow: reference # store the full payload, return an id to re-fetch

This prevents the case where a list endpoint returns complete objects because nobody wrote a projection, and forty of them land on turn six. The tool boundary is also the only place truncation is safe. Middleware that truncates the assembled prompt can cut a JSON payload in half and hand the model a fragment it will confidently misread.

3. Summarize with a summarizer you have tested

Compression is a prompt, and an untested prompt loses whatever it was not told to keep. Each strategy loses something predictable, so pick the loss you can live with.

Test it the way you test the agent. Take a transcript where a constraint was set at turn three, run the summarizer over it, and assert the constraint survives in the output. Keep that as a regression case and re-run it whenever the summarizer prompt or its model changes, because a summarizer is the one component whose failures look exactly like ordinary forgetting.

Pinning is what makes the remaining loss acceptable. Keep a small block outside the summarizable region entirely: constraints the user stated, identifiers, and decisions already made, assembled from structured fields rather than from prose a summarizer might rewrite. A constraint in the pinned block cannot be summarized away, since the summarizer never receives it as input.

4. Move durable facts to memory or records

Anything still true after the conversation ends does not belong in the transcript. Account identifiers, plan tier, stated preferences and settled decisions belong in a store, read into the pinned block at assembly time and written back when they change. The read and write model for this is covered in agent memory.

Scope the read as tightly as the write. A memory lookup that returns thirty loosely relevant items has relocated the context problem rather than fixed it, so cap how many memories are injected and rank them against the current turn instead of against the whole conversation.

5. Cap the transcript with an eviction rule you can state in one sentence

Two caps, whichever hits first: a token cap on the transcript slot and a turn cap. When either is hit, evict oldest-first until the slot fits and feed what you evicted to the summarizer.

One rule here is not negotiable. Never evict an assistant message containing a tool call without evicting its matching tool result, and never the reverse. Most providers reject a request with an orphaned tool call, so naive oldest-first eviction produces a hard 400 in the middle of a conversation that worked a minute earlier. Evict the pair, or evict only at turn boundaries.

6. Degrade on purpose at the ceiling

Decide the drop order before you are at the limit. A workable order is retrieved documents first, then oldest transcript pairs, then a forced re-summarization, with the pinned block and the output reservation excluded from all of it.

When that still leaves the request over the ceiling, end the turn with a stated message rather than truncating. Telling the user the conversation has reached its length limit and offering to continue in a fresh session with their current settings carried over is a worse answer than a full one and a far better answer than a mangled prompt or a provider error they read as a crash.

Why the middle of a long context gets ignored

Attention over a long input is uneven. Liu and colleagues measured it in Lost in the Middle, a 2023 study of two tasks, multi-document question answering and key-value retrieval: accuracy was highest when the relevant passage sat at the beginning or the end of the input, and degraded significantly when the model had to reach for it in the middle. A constraint stated at turn three of a forty-turn transcript sits squarely in that middle, so the transcript can fit inside the window with room to spare and the constraint can still be ignored.

Position is something you control at assembly time. Re-assert active constraints at the end of the assembled prompt, immediately before the current user message, rather than trusting the position they originally occupied. Published position studies measure whether a fact gets retrieved, and whether a stated rule gets obeyed at the same depth is a separate question with no public measurement behind it, so run that one on your own stack: put a known constraint at several depths, ask a question that depends on it, and count how often the answer honors it.

When to end the session instead of compressing it

Compression has a floor. Once the summary has been regenerated three or four times it is a summary of summaries, and further compression buys turns at the cost of the detail those turns were about.

Four signals say to end rather than compress: the task has changed, the summary has been regenerated more than twice, the agent re-asked something it was already told, or cost per turn has roughly doubled since the early turns. Ending well means carrying a handoff object into the next session, the pinned block plus a short statement of the current task, rather than starting from nothing and asking the user to repeat themselves.

Work that genuinely has to outlive a single conversation belongs in a different shape, described in long-running agents. The wider set of decisions about how an agent is assembled, from history handling to delegation and step budgets, sits in agent orchestration.

Where this gets easier

Most of the work above is choosing defaults and then keeping them visible six months later. Runtype makes those choices configuration rather than code: per-turn budgets are explicit settings (maxToolCalls per turn, loopConfig.maxTurns per run, a per-turn wall-clock budget, and an optional per-run cost ceiling), long-term memory is opt-in and keyed per agent, tenant or end user, and durable facts can live in records and collections instead of in the transcript. Execution traces carry per-step input and output with cached and uncached token counts separated, so a conversation that degraded at turn thirty is a setting you inspect rather than an emergent mystery.