RTPurbo transforms full-attention LLMs into sparse models by exploiting intrinsic sparsity through head-wise attention and dynamic top-p thresholding. Achieves near-lossless accuracy with only ~600 training steps, delivering up to 9.36× prefill and 2.01× decoding speedups.
论文原始摘要
Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-$p$ selection more suitable than fixed top-$k$ sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36$\times$ prefill speedup at 1M context and about a 2.01$\times$ decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.
Paper Collector 中文速览
仅用数百步训练将全注意力转换为高效稀疏注意力
方法概述
基于三个观察设计：仅保留检索头的完整KV缓存；用16维索引器实现高效长程检索；采用动态top-p选择替代固定top-k。通过利用模型内在稀疏性，RTPurbo在数百步训练内实现稀疏化。
核心贡献
发现全注意力LLM内在稀疏性，提出RTPurbo方法，仅需少量训练即可实现高效稀疏推理，保持近无损精度
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
How can full-attention LLMs be transformed into efficient sparse attention models with minimal training, leveraging the intrinsic sparsity already present in pretrained models?
核心方法
RTPurbo identifies retrieval heads through offline calibration using needle spans, applies different attention patterns to retrieval versus local heads, and uses low-dimensional pre-RoPE projections for efficient token selection with dynamic top-p thresholding. A two-stage training pipeline trains projection weights via KL divergence minimization, followed by self-distillation to align sparse model predictions with the dense teacher.
方法组件
- RTPurbo is a head-wise attention framework with precise token-level sparse computation.
- The method includes offline calibration for identifying retrieval heads.
- A two-stage training pipeline is required for the framework.
- Hardware-aware decoding kernel is provided for efficient implementation.
论点验证
The head-wise framework with token-level sparse computation is fully specified in Section 3. The dynamic top-p selection provides precise token-level control, demonstrated through experiments showing query-dependent token budgets (Table 6).
The custom GPU kernel is described in Section 3.4 with specific design details addressing both challenges. Performance is benchmarked in Figure 7 showing consistent speedup over FA2 and PyTorch implementations.
The claim references Table 1 for the comparison between top-k and top-p, but Table 1 is not provided in the available text. While the paper discusses the fixed-budget variant performing poorly on RULER 64K, the direct comparison data is missing.
Specific quantitative evidence is provided: paragraph 55 states 'the actual number of label tokens involved in learning is only about 1.2M' and 'about 600 steps' for convergence.
Specific quantitative evidence from paragraph 43 and Figure 1: prefill speedup from 2.83× at 32K to 9.36× at 1M; decode speedup from 1.47× at 32K to 2.01× at 1M.
This finding is attributed to prior work [19,28] and supported by the paper's own calibration results showing distinct retrieval vs local head behaviors.
Directly supported by Figure 3 and Table 6 showing different token budgets for different queries within the same head (468.8 for niah-S vs 2462.1 for multi-K at 32K).
The claim references Table 1 for the specific comparison data, but Table 1 is not provided in the available text. The quantitative claim cannot be verified without the referenced table.
The claim that calibration on 'one single long text sequence is sufficient' lacks experimental validation. No comparison is shown between partitions derived from single vs multiple calibration sequences.
Specific quantitative evidence from Table 5 (paragraph 40): RTPurbo achieves 86.67 on AIME, matching the dense baseline exactly.
Specific quantitative evidence from Table 6 (paragraph 41): 468.8 active tokens for niah-S vs 2462.1 for multi-K at 32K, demonstrating the 5× variance.
Specific quantitative evidence from Table 6 (paragraph 41): attention mass >0.93 and sparsity up to 89.2% at 64K.
... 共 54 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code available - implementation details of RTPurbo framework not accessible
- No training data available - datasets used for training and evaluation not specified
- Hyperparameters missing - learning rate, batch size, optimizer settings, exact number of training steps not provided
- Random seeds not specified for reproducibility
- Offline calibration procedure details missing - how retrieval heads are identified
- Two-stage training pipeline details not specified - what each stage involves
- Sparse computation pattern implementation details missing
- Hardware-aware decoding kernel implementation not provided
- Base model configurations and sizes not specified
- Evaluation tasks/metrics within lm-eval framework not detailed
局限与证据边界
- Our method relies on the empirical observation that attention heads can be partitioned into retrieval and local groups through offline calibration. While this behavior is stable in the models we study, the quality of this partition may degrade for models with weaker head specialization or under substantial domain shift.
- In the current design, retrieval heads still use full dense attention during prefill.
- Our experiments mainly focus on the Qwen3 family and on long-context and reasoning workloads, so broader validation on other architectures and domains is still needed.
- We expect these limitations to be addressed by future work on more adaptive head routing, stronger prefill sparsification, and broader cross-model evaluation.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-25T01:26:26+00:00 · 数据来源：Paper Collector