Context Rot: Why Bigger Context Windows Are Not the Answer

The arrival of million-token context windows inspired a tempting idea: just put everything in. In practice it fails badly. Models degrade long before their windows fill, and understanding why changes how you build.

When context windows expanded from a few thousand tokens to a million and beyond, many assumed the context problem was solved. Why engineer carefully what you include when you can fit entire codebases, whole document libraries, complete conversation histories? The assumption is wrong, and the phenomenon that proves it wrong has a name: context rot.

What Context Rot Is

Context rot is the gradual decay of model quality as the context fills over a long task. It is not a hard failure at the window limit; it is a steady degradation that begins far earlier. Research from Chroma and others shows model accuracy declining as input length grows, even on simple tasks, with measurable effects appearing around 32K tokens, a small fraction of modern window capacity.

The Evidence

The data is consistent across studies. A Databricks study found accuracy dropping around 32K tokens, long before any million-token limit. Anthropic reported that agent sessions stopping at 75% context utilisation produced higher-quality, more maintainable output than those pushed to the limit. The pattern is clear: more context in the window does not mean more capability; past a point it means less.

Why It Happens

Several mechanisms drive rot. The "lost in the middle" effect means information buried in a large context gets little attention. Accumulated history can distract the model from its training (the distraction failure mode). More content means more chances for irrelevant material to confuse or contradictory material to clash. The larger the context, the more these effects compound. A bigger window does not avoid them; it just provides more room for them to occur.

The Implication for Building

Context rot reframes the entire discipline. The goal is not to maximise how much you put in the window; it is to curate the smallest, highest-quality context that fully covers the task. Quality comes from curation, not capacity. This is why the four strategies exist: Write to offload, Select to include only what matters, Compress to keep it lean, Isolate to prevent interference. All four fight rot.

The Practical Rule

Treat the context window as a budget to spend wisely, not a container to fill. Aim to stay well within comfortable utilisation rather than pushing toward the limit. When context grows, do not celebrate that you have room to spare; ask what can be compressed, offloaded, or dropped. The teams building reliable AI in 2026 understand that the window size is almost irrelevant to quality. What matters is the discipline applied to what goes in it.