How to retain context

Every LLM application eventually runs into the same wall. The demo works. Then the conversation gets long, or the documents get big, or the agent takes twenty tool-call turns, and the thing starts forgetting instructions it followed perfectly five minutes ago.

The instinct is to reach for a model with a bigger window. That helps less than you’d hope, and this post is about what to do instead.

What a context window actually is

A model is stateless. It has no memory of your last request; each call is a fresh forward pass over whatever tokens you hand it. (If that sentence needs unpacking, the previous post covers the mechanism.)

The context window is the maximum number of tokens a single call may span — and it covers everything: the system prompt, the tool definitions, the whole conversation transcript, any documents you’ve pasted in, every tool result, and the output the model is currently generating. Input and output share the budget.

Two things people get wrong about it:

It’s measured in tokens, not words or characters. Prose runs somewhere in the region of three-quarters of a word per token in English, but code, JSON, identifiers and non-English text tokenise far less efficiently. A blob of minified JSON is dramatically more expensive than its character count suggests.

And it’s a hard ceiling, not a soft one. Cross it and the request fails or gets truncated. Windows across current models range from tens of thousands of tokens to millions, but the number is a limit, not a target.

What actually fills it

When people picture context exhaustion they picture a long chat. In practice, the conversation is rarely the problem. The offenders, roughly in order:

Tool results. One unbounded database query, one full file read, one HTTP response returned verbatim, and you’ve spent more context than the entire conversation preceding it. This is the number one cause and it’s the easiest to fix.

Retrieved documents. Ten chunks at a thousand tokens each, injected on every turn, is ten thousand tokens per turn.

Tool definitions. They sit in the system prompt on every call. Thirty tools with verbose schemas and examples is a permanent tax paid before the user has said anything.

Reasoning traces. With reasoning models, the thinking tokens are real tokens and they count.

The transcript itself, last.

Instrument this before you optimise it. Count tokens per component per turn and log it; nearly every team that does is surprised by what they find.

Why long contexts degrade

The window not erroring isn’t the same as the window working. Quality falls off well before the limit, for a few distinct reasons worth separating.

Dilution. Attention distributes a fixed budget of weight across all positions. The more tokens compete, the less any single instruction commands. Your carefully-worded system prompt is now one voice in a crowd of forty thousand.

Position effects. Models are consistently better at using information near the start and the end of a long input than material buried in the middle — the “lost in the middle” effect. It’s well-attested across models, and it means where you put something matters, not just whether it’s present.

Staleness and contradiction. Long agent runs accumulate superseded facts: a file read before it was edited, a plan that was abandoned, a failed approach retried. The model has no way to know which version is current. It sees two contradictory statements with equal authority and picks one.

Distance from the instructions. By turn thirty, the system prompt is thirty thousand tokens behind the cursor, and the local statistics of the transcript exert more pull than it does.

The through-line: context is a precision problem, not a recall problem. Irrelevant tokens are not neutral. They actively compete.

The techniques that work

Compaction

When the transcript crosses a budget, replace the old middle of it with a summary generated by the model itself, and keep going.

Don’t summarise naively. Decide what must survive verbatim — the original task, hard constraints, decisions already made, file paths and identifiers — and prompt the summariser to preserve those under fixed headings while compressing narrative. Then keep the last few turns raw, because recency is where the actual work is.

const SOFT_LIMIT = 60_000; // tokens; well under the hard ceiling

async function compact(messages: Message[]): Promise<Message[]> {
  if (countTokens(messages) < SOFT_LIMIT) return messages;

  const keep = messages.slice(-6);        // recent turns stay verbatim
  const older = messages.slice(0, -6);

  const summary = await model.complete({
    system:
      "Compress this transcript. Preserve exactly, under these headings: " +
      "TASK, CONSTRAINTS, DECISIONS MADE, FILES TOUCHED, OPEN QUESTIONS. " +
      "Discard conversational filler. Do not invent anything.",
    messages: older,
  });

  return [{ role: "user", content: `<summary>${summary}</summary>` }, ...keep];
}

Compaction is lossy by definition. That’s the point — but it’s also why the prompt above says do not invent anything, and why anything you cannot afford to lose belongs outside the transcript entirely.

Retrieval instead of stuffing

The scalable move is to stop putting the corpus in the window and start putting the relevant part of it in the window, per turn, on demand.

This is the whole argument for retrieval, and it’s why the next post is about vector databases and RAG. The short version: index your content once, fetch a handful of relevant pieces at query time, and spend your context on those instead of on everything.

Structured memory outside the window

Give the agent somewhere durable to write: a file, a table, a key-value store. Facts that must survive compaction get written there and read back on demand, addressed by key rather than by scrolling.

This inverts the default. Instead of “everything is in context until it falls off the end”, you get “context holds what this turn needs, and the store holds what the task needs”. It also makes state inspectable, which matters more than it sounds — you can read a memory file; you cannot read a 40k-token transcript with any confidence.

Discipline at the tool boundary

Most context bloat enters through tool results, so put the limits there:

• Paginate and cap. A tool that can return ten thousand rows should return fifty, with a cursor.
• Return references, not payloads. Ids, paths and line ranges; let the model ask for the body if it needs it.
• Filter server-side. Don’t return a document so the model can find one field.
• Give tools narrow schemas. A tool that can only do one thing needs less description.

A related point that isn’t about size at all: model output is untrusted input. A tool call is a string the model produced; it is not an authorisation. The AI assistant in my Money Diary app takes this literally — every tool call is validated with a Zod schema before it executes, and the userId it runs against is read from the session, never from the model’s arguments. The model can ask for anything it likes; it cannot ask on someone else’s behalf.

Prompt caching

Caching doesn’t reduce the tokens the model attends over, so it doesn’t help quality. It helps cost and latency, and it’s close to free to adopt — but only if you respect how it works.

Providers cache a prefix. A hit requires the beginning of your request to be byte-identical to a previous one. So:

• Stable content first: system prompt, tool definitions, long reference material.
• Volatile content last: the user’s turn, timestamps, anything per-request.
• Never interpolate a clock, a request id or a shuffled list into the top of the prompt. One changing character at position ten invalidates everything after it.

The natural shape of a conversation — a fixed prefix with new turns appended — is exactly the shape caching rewards. Most of the work is not accidentally breaking it.

Sub-agents

For genuinely large work, give a sub-task its own fresh window. It reads the twenty files, does the search, burns a hundred thousand tokens, and returns a paragraph. The parent’s context only ever sees the paragraph.

This is context isolation, and it’s the only technique here that lets total work exceed a single window without degrading the main thread.

Why a bigger window isn’t the answer

It’s an answer to exactly one problem — a single input that genuinely must be seen whole — and it’s the wrong answer to everything else.

Cost and latency scale with what you send, and you send it every turn. Attention is quadratic, so a very long prompt is not linearly more expensive to serve. And the quality curve doesn’t cooperate: adding marginally relevant material to a prompt reliably makes some answers worse, because you’ve increased the number of things competing for attention without increasing the number of things that matter.

The teams who get good results are not the ones with the largest windows. They are the ones who treat the window as a scarce, curated resource, and who can answer the question “what exactly is in this prompt, and why is each part of it there?”

Which turns the whole problem into a retrieval problem — and that’s next.