As reasoning chains grow longer, the model may not need to remember the entire history. Prefix Sliding keeps only the task prefix and a recent window, so long thinking no longer has to carry the full reasoning chain along.
Once a reasoning chain gets longer, how much does the model actually need to remember?
With standard full attention, the answer is: almost everything.
Previously computed values, intermediate steps, and the ever-accumulating reasoning history all remain within the scope of subsequent attention. The longer the reasoning, the higher the cost of generating each new token.
The question is: does the model really need to keep revisiting these past intermediate steps?
A recent joint study by Stanford University, the University of Washington, and other institutions tackles exactly this question, with Andrew Ng, Yejin Choi, and Percy Liang among the contributors.
The team proposes Prefix Sliding, which retains only the task prefix and the most recent reasoning window over the long run, allowing earlier intermediate tokens to gradually exit subsequent attention computation.
As a result, existing models achieve up to roughly 3x speedup on long thinking without retraining. When used in reinforcement learning, a single reasoning trajectory can extend beyond 100,000 tokens.
Paper title:
Prefix Sliding for efficient test-time scaling
Paper link:
https://arxiv.org/abs/2608.26070
Code repository:
https://github.com/Muennighoff/prefix-sliding
Why Attention Concentrates at Both Ends of Reasoning
The cost of full attention keeps rising with the length of the reasoning chain, but the model's use of these historical tokens is far from uniform.
After analyzing a reasoning trajectory of Qwen3-1.7B on AIME25, the team found that attention shows a clear "high at both ends, low in the middle" pattern.
The leading prefix and the most recently generated tokens receive the most attention, while the large mass of intermediate reasoning tokens sits at a consistently low level of attention.
In long reasoning, attention is mainly concentrated on the prefix and the most recently generated tokens.
The prefix holds the system instructions, task description, and tool information, and its very first tokens—especially the first four—also serve as attention sinks. The most recently generated tokens correspond to the problem the model is currently working on.
The paper uses a simple arithmetic example, ((42 + 84) × 4) - 5: once 42 + 84 is computed, the subsequent calculation may only need that result, without having to retain the full prior reasoning process.
This is the direct motivation behind Prefix Sliding: later computation may not need to carry the complete reasoning history at all times.
How Does Prefix Sliding Drop the Intermediate Tokens?
Prefix Sliding permanently retains the task prefix, while the most recent W reasoning tokens form a sliding window. As generation continues, earlier intermediate tokens are gradually removed from the KV cache used by subsequent attention computation.
Suppose the prefix has 100 tokens and the window is 4096. Even if the full reasoning has reached 100,000 tokens, subsequent attention computation only needs to keep roughly 4,196 tokens.
Once the window is full, the per-step cost no longer grows with the length of the full reasoning chain.
The prefix is permanently retained, while the most recent reasoning tokens move forward with the window.
A plain sliding window would gradually push the initial task information out of the window; Prefix Sliding keeps the prefix at all times.
The team ultimately adopts Continue PE. Experiments on AIME25 reported in the appendix show that it performs similarly to Reset PE, while avoiding the repeated application of new position encodings—it can directly reuse existing cached representations, making computation more efficient.
The team also implemented a dedicated FlashAttention kernel for NVIDIA Hopper.
Tiles lying entirely outside the prefix and sliding window are skipped outright, while tiles that partially overlap the valid region receive element-level masking, reducing unnecessary loads and computation. The resulting real-world speed is close to that of a plain sliding-window implementation.
Once the window is full, Prefix Sliding's generation throughput stabilizes.
Up to 3x Speedup Without Training; RL Extends to 100K Tokens
In the training-free setting, the authors use Qwen3-1.7B as the main model and evaluate on AIME25, GPQA, and MATH500.
Prefix Sliding's advantage comes from being able to generate more tokens within the same amount of thinking time—not from each individual token becoming better in quality.
With a 4096-token window, for example, Prefix Sliding matches full attention on AIME25, GPQA, and MATH500, but delivers markedly higher throughput on long sequences—5,224 tok/s at 128K, compared with just 448 tok/s for full attention.
Given the same thinking time, Prefix Sliding can generate more reasoning tokens.
The "3x" figure here compares thinking time at similar task performance, rather than directly comparing the throughput numbers above.
Given the same thinking time, Prefix Sliding can generate more reasoning tokens.
In the reinforcement learning phase, Prefix Sliding can also scale rollouts to over 100,000 tokens. To avoid full backpropagation over the entire ultra-long trajectory, the authors adopt truncated backpropagation.
The theoretical receptive field of a sliding window spanning multiple layers can reach W×L, but the authors cite prior analysis showing that, due to the information bottleneck, the effective receptive field in practice is closer to about 1.5×W—so when backpropagating on the final window, only a limited amount of preceding context is needed.
Ultra-long reasoning trajectories can use chunked or truncated backpropagation.
For example, on a 100,000-token trajectory with a window of 2048, the training side receives only the last 8,192 tokens: the first 6,144 serve as context, and the RL loss is computed on the final 2,048 tokens.
Experiments show that keeping only 1x the window leads to a large KL divergence; it drops significantly at 2x, and at 4x it is essentially on par with 8x—so the main experiments use the 4x window.
With more historical tokens fed in, the KL divergence of truncated backpropagation drops quickly.
The appendix also reports a smaller-scale asynchronous RL experiment on DeepSeek-R1-Distill-Qwen-7B.
The results show that with 32,768 tokens fed into training and Prefix Sliding backpropagating only over the last 8,192 tokens, training reward and AIME24 performance are on par with full attention.
On the 7B model, truncated backpropagation matches full attention.
The authors also clearly note that performance at larger scales and with longer reasoning chains still requires further validation.
Under a comparable memory budget, full attention supports a maximum length of 8,192 tokens, while Prefix Sliding progressively raises the maximum generation length to 104K and achieves a higher training reward.
Under a comparable memory budget, Prefix Sliding supports longer reinforcement learning trajectories.
Not All Intermediate Tokens Can Be Dropped
Compared with Last-k, Summary, and the plain sliding window, Prefix Sliding delivers the best overall combination of performance and efficiency.
Last-k repeatedly processes the retained tokens, while a plain sliding window gradually loses the task prefix. Summary also requires generating an extra summary and introduces more hyperparameters; in real examples, the model even appears to ignore the derivations already present in the summary and starts solving the problem over again.
Performance–efficiency comparison of Prefix Sliding against Last-k, Summary, and the plain sliding window.
In training-free LiveCodeBench experiments, the model may first write code, then continue thinking through thousands of tokens of comments; by the time it returns to the code, the earlier content has already slid out of the window.
A window of at least 16,384 tokens is required to match full attention.
The authors also note that if Prefix Sliding is used for reinforcement learning, the model may learn to adjust this kind of comment-based reasoning behavior, allowing for shorter windows.
LiveCodeBench requires a window of at least 16,384 tokens to match full attention.
Short tasks may not benefit either. HealthBench generates only about 2,086 tokens on average, and with a window of 2,048, many samples never truly enter the sliding phase, leaving little room for speedup.
In agent scenarios, a single overly long web page or file output can directly fill the limited window, to the point where the model may be unable to read the full content at once.
Multi-turn interactions also introduce new context management challenges: whether subsequent user instructions should be merged into the prefix for permanent retention, or allowed to slide out of the window later, remains an open question that the paper does not settle.
Prefix Sliding also does not reduce the prefill-stage KV cache overhead caused by ultra-long prefixes.
The paper's direct empirical comparisons are also largely limited to methods that can be applied directly to existing pretrained Transformers while keeping the per-step generation cost bounded; it does not cover other model architectures, sub-quadratic-complexity methods, or hybrid sliding-window models.
Conclusion
Prefix Sliding further concretizes the efficiency problem of long reasoning as a context management problem: how long should already-generated historical information be retained?
Which information should be permanently kept in the prefix, and which can move out with the window, still calls for more refined context management strategies. Whether additional memory mechanisms are needed in coding tasks, agent scenarios with large tool outputs, and multi-turn interactions also remains unsettled.
Current results show that a fixed prefix plus a sliding window can significantly reduce the cost of ultra-long reasoning, but its applicability to larger models and more complex long-horizon tasks still awaits further validation.