The paper identifies a structural shortcut bias in on-policy self-distillation for reasoning tasks, where the per-token signal equals conditional pointwise mutual information that rewards shortcut tokens and penalizes deliberation tokens.
论文原始摘要
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.
Paper Collector 中文速览
提出反向自蒸馏方法，加速数学推理能力提升
方法概述
通过点互信息分析发现特权上下文导致教师分布畸变，提出AntiSD通过最大化而非最小化师生分布差异，配合熵触发门控自动停止机制，实现高效自提升
核心贡献
揭示自蒸馏在数学推理中的失败根源，提出反向自蒸馏方法，用熵触发门控替代传统自蒸馏，实现2-10倍训练加速
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
Why does standard on-policy self-distillation fail for reasoning tasks, and how can the structural shortcut bias in its per-token signal be corrected?
核心方法
The authors derive that the per-token signal in self-distillation equals conditional pointwise mutual information between the next token and privileged context. They propose Anti-Self-Distillation (AntiSD), which reverses the gradient direction by ascending Jensen-Shannon divergence instead of descending, combined with an auto-calibrated entropy-triggered gate. Experiments train five language models (4B-30B parameters) on DAPO-Math-17k for 200 on-policy steps.
论点验证
This is a mathematical theorem proven in Appendix A, Lemma 2. The derivation uses Bayes' rule to show u_t = log(π_T/π_S) = PMI(y_t; c | x, y_{
This follows directly from the PMI interpretation. Since u_t = PMI(y_t; c | x, y_{
This is a direct logical consequence of claims 9 and 10, which are empirically and theoretically supported. The paper states this conclusion explicitly and it follows from the PMI analysis.
This observation (O2) is empirically demonstrated in Figure 2(b), which shows the asymmetric distribution with a heavier deliberation lobe. The paper notes individual tokens reaching u_t ≤ -20 in the tail.
This design choice is clearly motivated by observation O1 and implemented in AntiSD. The paper explains that reversing the gradient direction flips the per-token reward sign, addressing the wrong polarity issue.
This design choice is motivated by O2 and mathematically justified. Lemma 5 proves φ(u) ≥ -1/2 log 2, providing the asymmetric bound. The paper explains how JSD's shape counterbalances the empirical asymmetry.
The entropy-triggered gate is clearly described as a stabilizer for the ascent direction. Section 3.2 explains that once ascending a divergence, the policy gradient is no longer self-terminating, requiring a signal-quality criterion.
The gate mechanism is clearly specified with the Schmitt trigger design (disable at τ_down, re-enable at H_warm) to avoid chatter. This is a well-defined design choice.
The experimental setup is clearly specified: five models (Qwen3-8B, Qwen3-4B-IT-2507, Olmo3-7B-IT, Olmo3-7B-Think, Qwen3-30B-A3B), DAPO-Math-17k dataset, 200 on-policy steps. This is a verifiable experimental configuration.
... 共 41 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available - Algorithm 1 implementation details not accessible
- No training data (DAPO-Math-17k) available or link provided
- Specific model versions/checkpoints from Qwen3 and Olmo-3 families not specified (only parameter ranges 4B-30B mentioned)
- Learning rate not specified
- Batch size not specified
- Random seeds not reported for reproducibility
- Hardware specifications (GPU type, memory, number of devices) not mentioned
- GRPO hyperparameters beyond λ=0 not provided (optimizer, weight decay, etc.)
- Rollout group size for on-policy sampling not specified
- Temperature and other sampling settings referenced in Appendix but not accessible
局限与证据边界
- The PMI characterisation describes individual gradient contributions rather than the global optimum of the combined objective.
- Our evaluation focuses on math reasoning, leaving extensions to multi-turn agentic settings and broader coding benchmarks as natural next directions.
- The conditional-PMI account in Section 3.1 is a local, per-step characterization of the per-token signal rather than a global-optimum statement about the combined objective; understanding the long-horizon dynamics under the full ascent + gate update is itself an interesting question.
- Our evaluation spans five language models from the Qwen3 and Olmo-3 families (4B-30B parameters) on mathematical reasoning, with an initial probe on code reasoning.
- Larger model scales beyond 30B and multimodal conditioning are also natural settings to test whether the conditional-PMI characterization remains the dominant credit-assignment signal.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-21T13:30:56+00:00 · 数据来源：Paper Collector