Nvidia researchers say the million-token context window is mostly wasted money. In a technical report posted to arXiv on 3 November, a 12-author team argues that frontier models lose little or no accuracy when an agent keeps its working context to roughly 16K to 50K tokens, and that a model reading a long document in small pieces does better than one handed the whole thing.
The paper is titled “You don’t need long context” (arXiv 2611.05306). Its authors include Peter Belcak, who submitted it, and Pavlo Molchanov, and it links to an NVIDIA Research project page called Project Moonshot: Local Frontier Intelligence. It runs to 31 pages and eight figures. Its answer to its own root question, whether long contexts are necessary, is no.
Why length costs money
The first argument is about price. The cost of generating a token grows with how deep it sits in the context, because every new token attends to everything before it. Retail prices hide this, since providers quote one rate per million tokens whatever the window.
The team measured the gap on three open-weight models running on H100 GPUs, priced at US$3.85 per GPU-hour. Serving GLM 5.2 and Kimi K2.6 on 16 GPUs and Qwen 3 Coder Next on eight, a 256K window cost 3.6x, 5.6x and 7x more per generated token than a 16K window. On an NVIDIA DGX Spark, the time between tokens roughly doubled for several local models as the input got longer.

Windows have also outgrown the things people feed them. The Old Man and the Sea runs to about 37K tokens and The Great Gatsby to about 64K, yet the models in the paper’s survey ship with windows from 200K to about 1M.
How the tests worked
The team compared three ways of running the same agent. In every one, the model works in a plain loop that issues a single Bash command at a time.
| Regime | What the model sees |
|---|---|
| Full-history accumulation | Every action and result stays in one conversation. This is the baseline. |
| End-of-conversation compaction | Once the context crosses a threshold, the model writes a bounded brief of its state and a fresh conversation starts from the task plus that brief. |
| Single-action compaction | Each fresh conversation gets the task and the current state, takes exactly one action, updates the state and is discarded. |
For long documents, the paper doesn’t paste the input into the prompt. It saves each document as a file in the agent’s environment and lets the agent search it with ordinary tools. The agent greps for a name, number or phrase, then reads a window of roughly 6,000 characters around the hit. In one recorded LongBench v2 run, the agent translates a Kalamang sentence using a grammar of roughly 1M characters. It greps for one word, gets a line number and reads 36 lines.
grep -n -i -E '\bkademor\b' "$LONGBENCH_DOC" | head -20
doc-read --lines 3835:3870 "$LONGBENCH_DOC"
The benchmarks split into six long-horizon tasks (SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.1, FRAMES, τ²-bench and BFCLv3) and seven long-input ones (LongBench v2, LOONG, AA-LCR, Needle in a Haystack, RULER, Counting-Stars and BABILong). The principal runs used two closed-weight frontier models. The numbers below are Claude Opus 4.5’s.
What it found
On long-horizon tasks, the team stepped the total context limit down as low as 8K tokens, depending on the benchmark. Opus 4.5’s accuracy against its full-history baseline ranged from 3.7 points worse to 5.6 points better, and no more than 5 of 22 threshold settings lost more than 2 points. The best settings beat the baseline outright: 74.8% on SWE-bench Verified at a 24K limit against 70.0%, and 52.8% on Terminal-Bench at 32K against 47.2%.
Long documents went better still. On LongBench v2, Opus 4.5 scored 67.6% at a 16K limit against 68.0% for the full input. On LOONG, which asks questions across several financial reports, legal cases or academic papers, the average score rose from 70.1 to 79.8 at a 48K limit, and perfect answers from 38.9% to 52.6%. On AA-LCR it went from 65.0% to 84.0% at 16K. Across all seven long-input benchmarks and settings, Opus 4.5’s change ranged from 0.4 points worse to 40.0 points better. On Needle in a Haystack, both models reached 100%, which the authors put down to the questions sharing words with the needles.
65.0% to 84.0%
Opus 4.5 on AA-LCR at a 16K limit
US$52 to US$5.30
Estimated cost of that AA-LCR run
−0.4 points
The one small dip, on LongBench v2
The dollar figures are modelled, not billed. The authors take the 16K-to-256K cost ratio from their open-weight measurements, round the mean to 0.20, apply it to Opus 4.5’s list prices of US$5 per million input tokens, US$0.50 cached and US$25 output, and interpolate from a 200K reference window. On that basis the AA-LCR run drops from US$52.00 to US$5.30 and the LOONG run from US$1,367.50 to US$181.11. Their stated savings for Opus range from 2.2x to 8.8x on long-horizon tasks and 6.5x to 54.6x on long-input ones.

One action per context
The most extreme test cut the context to almost nothing. Under single-action compaction, each fresh conversation gets the task and a written state, runs exactly one command, updates the state and is thrown away. On Opus 4.5 this scored 70.2% on SWE-bench Verified against 70.0% for full history, and 51.8% on Terminal-Bench against 47.2%. The median peak context fell from 32K to 8K tokens on SWE-bench Verified and from 27K to 9K on Terminal-Bench. The largest single call fell from 200K to 16K and from 130K to 19K.

A compaction that works isn’t a summary. The team found that summaries give every fact equal weight and lean on repeated stretches of the agent’s trace, whatever their relevance. The compactions that held accuracy best carried a near-verbatim copy of the task, the agent’s current intentions and next steps, and the artefacts that future work needs, such as a decomposed plan, a list of completion criteria and pointers to files.

The model class they propose
From this the authors outline a new model class they call semantic state space models, or SSSMs. An SSSM is post-trained to read a written description of a conversation’s state, reason about the next action, take it, register the outcome and rewrite the state so that a separate instance of the model can carry on. Unlike a state space model such as Mamba, which passes a black-box neural state between tokens, an SSSM keeps its state in plain language between turns. The authors surmise that such a model would be cheaper to train, because it wouldn’t need to learn which patches of earlier context to attend to, and they see no reason it couldn’t match a model that accumulates full history.
Why train full-history-accumulating models in the first place?
Belcak et al., arXiv 2611.05306
What the paper doesn’t show
The paper trains no SSSM. The single-action results come from existing models prompted to compact, and every figure above comes from the authors’ own harnesses, a Bash-only loop built on mini-swe-agent and their own mini-agent. Compaction is lossy, and the authors say so: they build a coin-toss task that a fixed-size compaction is guaranteed to fail. Their fix is to keep every message in a file the agent can search, which turns remembering into a long-input task of the kind the paper already covers.
Our evidence does not rule out the existence of tasks for which long context is beneficial, nor does it prove that every possible task can be solved with short context.
Belcak et al., arXiv 2611.05306, conclusion
For anyone building agents, the practical content is the harness. Store long inputs as files, search them with grep, and compact with a brief that carries the task, the plan and the pointers.
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.




