TL;DR
This paper identifies two conditions governing On-Policy Distillation effectiveness: thinking-pattern consistency and new knowledge provision. Successful distillation is driven by progressive alignment on shared high-probability tokens carrying 97%-99% of probability mass, explaining why stronger t…
This paper identifies two conditions governing On-Policy Distillation effectiveness: thinking-pattern consistency and new knowledge provision. Successful distillation is driven by progressive alignment on shared high-probability tokens carrying 97%-99% of probability mass, explaining why stronger t…
35
已证实
3
证据不足
1
无法验证
N/A
可复现性
核心问题
What conditions govern the effectiveness of On-Policy Distillation (OPD) for large language models, and why do stronger teachers not always yield better distillation outcomes?
核心方法
{'approach': 'The authors conduct controlled experiments across multiple model families (Qwen, DeepSeek, R1-Distill), defining dynamic metrics (Overlap Ratio, Overlap-Token Advantage, Entropy Gap) to monitor training dynamics. They perform ablation studies including reverse-distillation experiments and targeted token-region optimization to isolate the mechanisms driving OPD success or failure.', 'key_components': ['The high-probability-token alignment phenomenon generalizes across different model pairs.', 'Successful OPD consistently coincides with emergence of high-probability-token alignment at student-visited states.', 'Failed OPD shows poor or unstable alignment metrics across different teacher-student configurations.'], 'section_ids': ['sec_36']}
论点验证
已证实 (80%)
We identify two conditions that jointly govern the outcome [of OPD]: thinking-pattern consistency and new knowledge beyond what the student has seen during training.
The paper provides substantial experimental evidence across Sections 3.1, 3.2, and the reverse-distillation experiments to validate both conditions. Multiple controlled comparisons demonstrate that thinking-pattern consistency (higher initial overlap
The paper provides substantial experimental evidence across Sections 3.1, 3.2, and the reverse-distillation experiments to validate both conditions. Multiple controlled comparisons demonstrate that thinking-pattern consistency (higher initial overlap
已证实 (95%)
We define the student's and teacher's top-k sets at step t as S(p)_t = TopK(p_t, k) and S(q)_t = TopK(q_t, k), respectively. The following metrics are monitored throughout OPD training: Overlap Ratio, Overlap-Token Advantage, and Entropy Gap.
The paper provides clear mathematical definitions for all three metrics (Overlap Ratio in p16-17, Overlap-Token Advantage in p18-19, Entropy Gap in p20-21) and uses them consistently throughout the experimental analysis. This is a methodological cont
The paper provides clear mathematical definitions for all three metrics (Overlap Ratio in p16-17, Overlap-Token Advantage in p18-19, Entropy Gap in p20-21) and uses them consistently throughout the experimental analysis. This is a methodological cont
已证实 (85%)
Distillation from Qwen3-4B-Base-GRPO consistently outperforms distillation from Qwen3-4B (Non-thinking), although the two teachers have broadly comparable performance.
The claim is supported by quantitative experimental results shown in Figure 2, with the paper explicitly stating that distillation from GRPO teacher 'consistently outperforms' the Non-thinking teacher. The paper also notes both teachers have 'broadly
The claim is supported by quantitative experimental results shown in Figure 2, with the paper explicitly stating that distillation from GRPO teacher 'consistently outperforms' the Non-thinking teacher. The paper also notes both teachers have 'broadly
已证实 (85%)
The GRPO teacher exhibits a higher initial overlap ratio, suggesting that its thinking pattern aligns more closely with the student.
The paper provides quantitative evidence from Figure 2 (right) showing the GRPO teacher has higher initial overlap ratio. This is a direct experimental observation.
The paper provides quantitative evidence from Figure 2 (right) showing the GRPO teacher has higher initial overlap ratio. This is a direct experimental observation.
已证实 (85%)
Same-pipeline teachers yield limited improvement, while the post-trained teachers produce substantially stronger gains across all benchmarks.
The claim is supported by quantitative results in Figure 4, with specific gap recovery rates reported. The paper shows post-trained teachers recover 'a much larger fraction of the teacher-student gap' compared to same-pipeline teachers.
The claim is supported by quantitative results in Figure 4, with specific gap recovery rates reported. The paper shows post-trained teachers recover 'a much larger fraction of the teacher-student gap' compared to same-pipeline teachers.
已证实 (85%)
Distilling JustRL-1.5B toward R1-Distill-1.5B, its own pre-RL checkpoint, causes the student to regress almost exactly to its pre-RL performance, removing all gains acquired through RL.
The paper provides quantitative evidence from Figure 5 showing that distilling JustRL-1.5B toward its pre-RL checkpoint causes regression to pre-RL performance. This is a specific experimental result.
The paper provides quantitative evidence from Figure 5 showing that distilling JustRL-1.5B toward its pre-RL checkpoint causes regression to pre-RL performance. This is a specific experimental result.
已证实 (85%)
When we replace the teacher with R1-Distill-7B, a substantially larger and even slightly stronger model from the same family, the training trajectory... produces no improvement and instead causes regression.
The paper provides experimental evidence from Figure 5 showing that using R1-Distill-7B as teacher produces no improvement and causes regression, despite it being a larger and slightly stronger model.
The paper provides experimental evidence from Figure 5 showing that using R1-Distill-7B as teacher produces no improvement and causes regression, despite it being a larger and slightly stronger model.
已证实 (85%)
Distillation from JustRL-1.5B yields consistent gains, with the final student recovering more than 80% of the performance gap to the teacher, whereas distillation from R1-Distill-7B fails to yield any improvement despite the teacher being stronger overall.
The paper provides specific quantitative results: 'recovering more than 80% of the performance gap' for JustRL-1.5B teacher vs 'fails to yield any improvement' for R1-Distill-7B teacher, with reference to Figure 6.
The paper provides specific quantitative results: 'recovering more than 80% of the performance gap' for JustRL-1.5B teacher vs 'fails to yield any improvement' for R1-Distill-7B teacher, with reference to Figure 6.
已证实 (85%)
In the successful run, the overlap ratio rises steadily, the overlap-token advantage improves toward zero, and the entropy gap narrows, indicating that the student progressively locates the teacher's high-probability region, calibrates its mass within that region, and matches the teacher's local confidence. In the failing run, all three metrics stagnate.
The paper provides detailed observations of training dynamics from Figure 6 (bottom), describing specific metric behaviors in successful vs failing runs. These are quantitative observations from tracked metrics.
The paper provides detailed observations of training dynamics from Figure 6 (bottom), describing specific metric behaviors in successful vs failing runs. These are quantitative observations from tracked metrics.
已证实 (85%)
The overlap tokens carry 97%-99% of the total probability mass for both models throughout training, so the rising overlap reflects alignment on the probabilistically dominant tokens, not merely a set-level coincidence.
The paper provides specific quantitative measurements: '97%-99% of the total probability mass' with reference to Appendix B.1 and Figure 18. This is a direct measurement from the experiments.
The paper provides specific quantitative measurements: '97%-99% of the total probability mass' with reference to Appendix B.1 and Figure 18. This is a direct measurement from the experiments.
已证实 (85%)
Optimizing only the overlap region is sufficient to recover nearly the full benefit of standard Student Top-k OPD on all three benchmarks, while Non-Overlap Top-k remains consistently weaker.
The paper provides ablation study results from Figure 7 showing that Overlap Top-k achieves nearly the same performance as Student Top-k across all three benchmarks, while Non-Overlap Top-k is consistently weaker.
The paper provides ablation study results from Figure 7 showing that Overlap Top-k achieves nearly the same performance as Student Top-k across all three benchmarks, while Non-Overlap Top-k is consistently weaker.
已证实 (85%)
Both Student Top-k and Overlap Top-k raise the overlap ratio steadily from about 72% to above 91%, while Non-Overlap Top-k first decreases and then only partially recovers.
The paper provides specific quantitative values from Figure 7: overlap ratio rises from ~72% to above 91% for Student Top-k and Overlap Top-k, while Non-Overlap Top-k shows different behavior.
The paper provides specific quantitative values from Figure 7: overlap ratio rises from ~72% to above 91% for Student Top-k and Overlap Top-k, while Non-Overlap Top-k shows different behavior.
已证实 (80%)
We present two complementary strategies that recover OPD in otherwise failing configurations by improving the overlap dynamics: off-policy cold start and teacher-aligned prompt selection.
The paper presents two strategies in Section 5 with experimental validation: off-policy cold start (Section 5.1, Figure 8) and teacher-aligned prompt selection (Section 5.2, Figures 9-10). Both strategies are shown to improve OPD outcomes in otherwis
The paper presents two strategies in Section 5 with experimental validation: off-policy cold start (Section 5.1, Figure 8) and teacher-aligned prompt selection (Section 5.2, Figures 9-10). Both strategies are shown to improve OPD outcomes in otherwis
已证实 (85%)
The two-stage approach substantially outperforms pure OPD. Starting from Qwen3-1.7B-SFT yields consistently better validation performance than starting directly from Qwen3-1.7B-Base.
The paper provides quantitative results from Figure 8 showing the two-stage SFT+OPD approach consistently outperforms pure OPD starting from base model.
The paper provides quantitative results from Figure 8 showing the two-stage SFT+OPD approach consistently outperforms pure OPD starting from base model.
已证实 (85%)
The SFT-initialized student begins with a much higher overlap ratio and maintains a smooth, stable trajectory, whereas the base-initialized student starts lower and exhibits pronounced instability before gradually recovering.
The paper provides observations of overlap dynamics from Figure 8, describing specific differences in trajectory stability between SFT-initialized and base-initialized students.
The paper provides observations of overlap dynamics from Figure 8, describing specific differences in trajectory stability between SFT-initialized and base-initialized students.
已证实 (85%)
Simply switching to the teacher-aligned template improves validation performance on all three benchmarks.
The paper provides quantitative results from Figure 9 showing teacher-aligned template improves validation performance on all three benchmarks.
The paper provides quantitative results from Figure 9 showing teacher-aligned template improves validation performance on all three benchmarks.
已证实 (75%)
Using teacher-aligned prompts leads to substantially lower student entropy during training. This suggests that performing OPD only on prompts seen during teacher post-training may not always be ideal, as it can overly reduce policy entropy.
The paper observes that teacher-aligned prompts lead to 'substantially lower student entropy' (quantitative observation from experiments). However, the 'suggestion' that this may not be ideal is an interpretation rather than a rigorously proven findi
The paper observes that teacher-aligned prompts lead to 'substantially lower student entropy' (quantitative observation from experiments). However, the 'suggestion' that this may not be ideal is an interpretation rather than a rigorously proven findi
已证实 (90%)
The teacher's accuracy advantage decreases monotonically, from +0.37 at a 1K prefix to just +0.02 at a 16K prefix.
The paper provides specific quantitative measurements from Figure 11(b): teacher's accuracy advantage decreases from +0.37 at 1K prefix to +0.02 at 16K prefix.
The paper provides specific quantitative measurements from Figure 11(b): teacher's accuracy advantage decreases from +0.37 at 1K prefix to +0.02 at 16K prefix.
已证实 (85%)
High entropy first appears at the end of the response and progressively propagates toward earlier tokens as training proceeds.
The paper provides observations from Figure 13 showing the back-to-front pattern of entropy propagation during training.
The paper provides observations from Figure 13 showing the back-to-front pattern of entropy propagation during training.
已证实 (90%)
For both teachers, correct rollouts consistently receive higher sequence mean reward than incorrect ones, with comparable AUROC values (0.73 for JustRL-1.5B, 0.75 for R1-Distill-7B). The failing 7B teacher does not produce a weaker global signal.
The paper provides specific quantitative AUROC values from Figure 14: 0.73 for JustRL-1.5B and 0.75 for R1-Distill-7B, showing both teachers produce globally informative reward signals.
The paper provides specific quantitative AUROC values from Figure 14: 0.73 for JustRL-1.5B and 0.75 for R1-Distill-7B, showing both teachers produce globally informative reward signals.
... 共 39 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available - implementation details are not accessible
- No data available - DAPO-Math-17K dataset and prompt templates not provided
- Training hyperparameters not specified (learning rate, batch size, epochs, optimizer, etc.)
- Distillation-specific parameters missing (temperature, loss function formulation, KL divergence weight)
- Random seeds not reported for reproducibility
- Hardware/environment specifications not mentioned (GPU type, memory, framework versions)
- Complete prompt templates referenced but not shown in the provided sections
- Section 4.1 referenced for training/evaluation setup but not provided in the excerpt
- Model architecture details beyond names not specified
- Evaluation metrics implementation details not provided
局限性（作者自述）
- Dense reward is effective on moderately long reasoning traces, but its reliability degrades with depth as the student prefix drifts further from the states familiar to the teacher. This suggests that OPD may not extend cleanly to longer-horizon settings such as extended chain-of-thought or agentic multi-turn interaction.
- We have not directly verified this anisotropy hypothesis, and doing so would require analyzing the directional structure of per-token gradients, which we leave to future work.
- These findings reveal a fundamental tension between supervision density and supervision reliability, and point to the limitations of current OPD for long-horizon reasoning and agentic settings.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-04-20T13:08:56+00:00 · 数据来源：Paper Collector