Context Rot: Why Your LLM Gets Worse the More You Give It

A 1M-token context window sounds like it ends the retrieval problem. Just put everything in and let the model sort it out.

It does not work, and the reason has a name now: context rot.

What is context rot?

Context rot is the degradation in output quality as input length grows — even when you are nowhere near filling the context window.

That “even when” is the whole point. This is not the model running out of room and truncating. It is a model with 900,000 tokens of headroom still getting measurably worse because you handed it 100,000 instead of 8,000.

The phrase started circulating in June 2026, off the back of Chroma’s research, which evaluated 18 models — GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 among them. The finding that mattered: models do not use their context uniformly. Performance gets steadily less reliable as input grows, and how fast it degrades varies a lot between models.

For the 1M-token models, a clearly observable effect typically starts somewhere around 300,000–400,000 tokens. Well short of the advertised limit.

It is not the same as “lost in the middle”

These two get used interchangeably and they are different problems with different fixes.

Lost in the middle is positional. Accuracy depends on where the relevant fact sits in the window, and it follows a U-shape: strong at the beginning, strong at the end, and 20–30 points lower in the middle. Move the same fact to the top and accuracy recovers.

Context rot is about length. Accuracy declines as the input grows even when the evidence is fixed and favourably placed. Moving the fact does not save you, because the problem is not where it is — it is everything else you sent alongside it.

You can have both at once. If you only ever test by burying a fact in the middle, you will diagnose the wrong one.

Why the benchmark misled everyone

Most long-context claims lean on Needle in a Haystack — hide a sentence in a long document, ask the model to find it. Models score near-perfectly, vendors publish the graph, everyone concludes long context is solved.

NIAH mostly tests lexical retrieval. The needle is verbatim; matching it is close to string search. Real work asks for something harder: read forty documents that all sort of relate to the question, and synthesise. That is semantic, and it is where the numbers fall apart.

A good rule: if your evaluation would pass with grep, it is not measuring what your product does.

What this means for RAG

The instinct when a RAG system misses an answer is to widen retrieval — top-k from 5 to 20, chunk size up, “give it more to work with.”

Context rot says that instinct is often backwards. Going from 5 chunks to 20 adds one relevant chunk and fifteen near-misses, and those fifteen are not free. They compete for the same attention.

Which reframes the job. You are not filling a window, you are spending a budget:

Retrieve wide, then cut hard. Pull 50 candidates so recall is high, then re-rank and keep 5. The wide net is for finding; the model only sees what survives.

Compress before you send. Summarise retrieved chunks down to the claims that bear on the question. A 200-token summary of a 2,000-token document is often strictly better input.

Put the critical material at the edges. Since the middle is measurably weaker, structure matters: instructions at the top, the most relevant evidence near the end, the bulk in between.

Measure it on your own data

Vendor context-length claims tell you nothing about your workload. This is cheap to check:

import statistics

def context_rot_curve(questions, retrieve, ask, grade, k_values=(3, 5, 10, 20, 50)):
    """Same questions, same evidence, increasing amounts of surrounding context.
    If accuracy peaks at k=5 and falls after, that is your budget — not the window."""
    curve = {}
    for k in k_values:
        scores = []
        for q in questions:
            chunks = retrieve(q, k=k)          # relevant chunk stays in every k
            answer = ask(q, context=chunks)
            scores.append(grade(q, answer))    # exact match, or an LLM judge
        curve[k] = statistics.mean(scores)
        print(f"k={k:>3}  tokens≈{sum(len(c) for c in chunks)//4:>6}  acc={curve[k]:.3f}")
    best = max(curve, key=curve.get)
    print(f"\npeak at k={best} — adding context past this point costs accuracy")
    return curve

Run it once. Most teams find the peak is much lower than what they shipped, and that they have been paying more per call for worse answers.

A prompt for auditing what you send

Before optimising retrieval, it is worth knowing what is actually in the window. This works well pasted alongside a real assembled prompt:

Below is the full context my RAG system sent to the model for one question,
plus the question itself.

QUESTION: [PASTE]
CONTEXT AS SENT: [PASTE THE WHOLE THING]

Analyse it as a context budget:

1. Which passages actually bear on the question? Quote the specific lines.
2. Which passages are near-misses — topically related, but not usable as
   evidence for this question? These are the expensive ones. List them.
3. What percentage of these tokens is doing no work?
4. Is the critical evidence near the start, the middle, or the end?
5. Rewrite the context to the smallest set that still supports a complete
   answer. Show it, and tell me the token count before and after.

Do not answer the question itself. I am auditing the input, not the output.

That question 2 is the useful one. Irrelevant chunks are easy to spot and easy to drop. Near-misses are what quietly wreck long-context performance, and they are exactly what a similarity search returns more of when you raise k.

The bottom line

A large context window is a capacity, not a strategy. Models degrade as input grows regardless of how much room is left, degradation starts far below the advertised limit, and the standard benchmark hides it because verbatim retrieval is far easier than synthesis.

Treat the window as a budget with a real cost per token, measure where your own accuracy peaks, and send the smallest context that can answer the question.

If you are choosing what goes into that budget in the first place, that is context engineering — and it is the discipline this whole problem belongs to.