There is a folk theory of the context window that quietly ruins prompts: the belief that it is storage — a bucket you fill, where anything dropped in is equally available later. It is not a bucket. It is a working set the model attends over, all at once, every time it generates a token. And attention is not free. Every token you add competes with every other token for a fixed pool of it. The only honest question, asked of each token, is whether it earns its place.
Start with what the window actually is. Not a memory the model keeps between turns — it keeps nothing. On every inference the harness rebuilds an ordered array of tokens from scratch and hands it over whole:
Order is transcript order. The system prompt sits at the head; turns append chronologically; the latest request lands at the tail, not the top. Generation happens at position n+1, attending backward over everything before it. Four kinds of token compete inside that array — instructions (system prompt, specs, role), retrieved knowledge (RAG, docs, memory), conversation history, and tool outputs — and the model cannot tell which kind you meant to matter.
The budget is arithmetic, not metaphor
Attention for each generated token is a softmax over all prior positions. The weights are non-negative and they sum to one — always, no matter how long the window gets:
This is the whole game in one line. The pool does not grow when you add context; it gets redivided. Every token you paste in dilutes every other token arithmetically. "More context" is not a free improvement — it is a transfer, taking budget from the tokens that mattered and handing it to the ones that didn't.
The middle is where information goes to die
Recall as a function of relative position — where a token sits, from head to tail —
is not flat. It is a bathtub: high at the head, high at the tail, and sagging across the vast middle.
The head keeps its privilege for a mechanical reason: the very first tokens act as an attention sink, and they are permanently occupied by the system prompt. Which sets up the trap. Content does not float toward the head as the conversation grows — its absolute position freezes while the tail races away from it:
A chunk born at the tail slides through the sag and parks in the shallow middle. It never gains head privilege, because primacy is anchored to the absolute first tokens, not to whatever is oldest.
Three ways context turns against you
Lost-in-the-middle. The mechanism above: a relevant fact buried at p ≈ 0.5 is attended to less than the same fact at either edge. Position is a lever you are almost always leaving unpulled.
Distraction — "worse than nothing." This is the counterintuitive one. A buried relevant document competes with plausible-but-wrong distractors that also claim budget, and the model can anchor on the confident-sounding wrong thing near a privileged position. With no documents it falls back on clean parametric knowledge and does better. Adding irrelevant-but-plausible context can measurably degrade reasoning. More is not more.
Staleness. Old or contradictory context does not sit there harmlessly — it silently poisons the output, because the model has no way to know that the thing you said forty turns ago has been superseded.
You cannot weight a token. So climb the ladder.
The obvious wish — "just make the model pay more attention to this" — is not available. Attention weights are computed, not exposed; no API lets you set them. What you have instead is a ladder of proxy levers, weakest to strongest: position (put it near an edge) → system-prompt privilege (the model is trained to defer to it) → salience markers (ALL CAPS, "IMPORTANT" — a learned convention, and inflationary: the more you use them the less each is worth) → repetition near the tail → and finally, stepping out of the attention game entirely, enforcement at the boundary: a schema, a validator, a hook. That last rung has effectively infinite weight, because it does not ask the model to comply — it refuses the output that doesn't.
The discipline
All of this collapses into one habit. For every token in the window, ask: keep it · retrieve it on demand · compress it · drop it. Clearing the session on a change of subject is the same move seen from the other side — old-subject tokens are irrelevant-but-plausible, near-zero signal, still billed against the budget. The refinement is not to clear blindly but to compress-then-drop: extract the durable kernel to a file, and start the next session with it at the head, where the attention sink will actually hold it. The boundary test is a single question — will the model need to attend to anything from the earlier chain? No, clear it; a little, compact it forward; pervasively, don't clear yet.
A context window is a working set, and every token in it is paying rent out of a budget that never grows. Engineering it well is mostly the willingness to evict.