This paper introduces a discriminator view of RLVR updates, showing that policy-gradient updates act as implicit linear discriminators over token-gradient vectors.
论文原始摘要
Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose $\textbf{DelTA}$, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.
Paper Collector 中文速览
提出DelTA方法改进可验证奖励强化学习的token级信用分配
方法概述
从判别器视角分析RLVR更新，发现其隐式作为token梯度向量的线性判别器。DelTA估计token系数来放大特定方向的token梯度并抑制共享或弱判别性方向，重加权自归一化RLVR代理目标，使有效的侧面质心更具对比性
核心贡献
提出DelTA判别式token信用分配方法，通过估计token系数重塑RLVR更新方向，增强高低奖励响应的区分度
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
Which token probabilities are increased or decreased by RLVR updates, and what determines these changes when response-level rewards create a granularity mismatch with token-level policy updates?
核心方法
The authors develop a discriminator view showing RLVR updates act as implicit linear discriminators over token-gradient vectors, with side-wise centroids determining token probability changes. They propose DelTA, which reweights token-gradient terms using discriminative scores computed via an entropy-regularized assignment problem, iteratively refining centroids to amplify side-specific directions while downweighting shared patterns.
方法组件
- RLVR induces sparse token-distribution changes that are not directly explained by the reward signal.
- Token-level selection effects are induced by how token-gradient vectors are aggregated in policy-gradient updates.
- The analysis uses DAPO as a concrete instance but extends to broader classes of sequence-level RLVR objectives.
- DelTA reweights token-gradient terms to reshape the induced discriminator and update direction.
- DelTA was tested on Olmo3-7B-Base using the same hyperparameters as main experiments without architecture-specific tuning.
- DelTA consistently outperforms DAPO on all seven mathematical reasoning benchmarks, improving average score from 19.01 to 22.80.
- Particularly large improvements were observed on AIME24, AIME26, HMMT25-Nov, and Brumo25 benchmarks.
- Results demonstrate that DelTA is not tied to a specific Qwen3 backbone and generalizes across different model families.
- Code generation experiments were conducted using Eurus2-RL-Code dataset with DAPO baseline, evaluating on HumanEval+, MBPP+, and LiveCodeBench.
论点验证
The paper provides the mathematical derivation showing this duality explicitly. Equation (4) in p_21 shows how the update direction defines scores that act as a linear discriminator over token-gradient vectors.
The DelTA method is fully specified in Section 3.2 with equations (5)-(8) and detailed algorithm in Appendix H. The method reweights token-gradient terms using discriminative scores.
This is a restatement of the discriminator view contribution. The mathematical derivation in Section 3.1 provides the evidence.
The method is fully specified with equations (5)-(8) showing how discriminative scores are computed and used to reweight the surrogate.
Specific quantitative results are provided in the text. The paper states DelTA improves average score from 25.14 to 28.40 on 8B (3.26 points) and provides the 2.62 point improvement on 14B.
The paper explicitly states this result and references Table 1 with benchmark-by-benchmark results showing DelTA achieves best results on every benchmark.
Specific numbers are provided: baseline 25.14, DelTA 28.40, improvement of 3.26 points.
The paper describes Figure 2 which shows training dynamics, explicitly stating that DAPO plateaus and slightly degrades while DelTA continues improving to higher final reward.
The paper describes Figure 2 showing response-length and entropy curves that demonstrate this behavior pattern.
The paper references Table 2 showing the withinside-only variant performs worse than both DelTA and DAPO baseline.
This is a valid interpretation of the ablation result showing the withinside-only variant underperforms, indicating own-side centrality alone is insufficient.
This is a conceptual conclusion drawn from the ablation. The reasoning is sound but it's an interpretation rather than directly measured.
Specific quantitative results are provided: default proxy 23.27, top-K proxy 24.29, showing robustness to proxy choice.
Specific quantitative result provided: random-λ baseline drops to 18.34 average score.
Valid interpretation of the random-λ baseline result showing arbitrary reweighting doesn't help.
Specific quantitative measurements provided: 37 seconds longer, 10.2% of total first-step time on 8 NVIDIA B200 GPUs.
Specific quantitative results provided for Olmo3-7B-Base: average improves from 19.01 to 22.80 (3.79 points gain).
Specific quantitative results provided: weighted average improves from 47.7 to 49.5 on code generation benchmarks.
... 共 42 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available
- No training data (DeepMath-103K) available
- Learning rate not specified
- Batch size not specified
- Number of training epochs/steps not specified
- Random seeds not provided
- Hardware specifications (GPU type, number of GPUs) not mentioned
- Optimizer type and settings not specified
- Numerical constants ε and ε_γ values not provided
- Evaluation sampling parameters (temperature, top-p, top-k) not specified
局限与证据边界
- DelTA estimates token coefficients using a layer-restricted token-gradient proxy rather than full-parameter token gradients, since computing full gradients for all sampled tokens is computationally expensive at RLVR scale.
- exploring richer and more efficient token-gradient approximations is a promising direction for future work.
- our empirical evaluation focuses primarily on mathematical reasoning, with additional validation on code generation, different backbone architectures, and out-of-domain benchmarks. Future work could further evaluate DelTA on broader RLVR settings, including multi-turn interaction, tool-use tasks, and domains with more diverse verifiable signals.
- DelTA introduces additional computation for coefficient estimation.
- future engineering improvements such as more efficient caching or lower-cost proxy computation could further reduce this cost.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-23T07:14:54+00:00 · 数据来源：Paper Collector