TL;DR
DVAO introduces dynamic variance-adaptive weighting for multi-reward GRPO in LLMs, addressing magnitude explosion and objective isolation. It bounds advantages while enabling cross-objective regularization, achieving superior accuracy-compliance trade-offs across tasks and scales.

论文原始摘要

Reinforcement Learning has become a standard paradigm for aligning Large Language Models with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.

Paper Collector 中文速览

方差自适应的多奖励优势优化方法

方法概述

DVAO根据rollout组内各目标的经验奖励方差动态调整组合权重，增强强学习信号目标权重并抑制噪声目标，通过数学证明保持有界优势幅度，引入自适应跨目标正则化机制

核心贡献

提出基于奖励方差动态调整权重的DVAO算法，解决多奖励RL中的训练不稳定性问题

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

26
已证实
9
证据不足
5
无法验证
N/A
可复现性
置信度
86%

核心问题

How can Group Relative Policy Optimization (GRPO) be effectively adapted to multi-reward settings for Large Language Models while avoiding the magnitude explosion of Reward Combination and the objective isolation of Advantage Combination?

核心方法

DVAO replaces fixed combination weights with variance-adaptive weights that up-weight high-variance objectives and down-weight low-variance ones within rollout groups. The authors provide theoretical proofs (Propositions 1-3) establishing bounded advantage magnitudes and cross-objective regularization properties, then evaluate on mathematical reasoning (AIME, MATH500, OlympiadBench, AMC23) and tool-use (BFCL-v4) benchmarks using Qwen models.

方法组件

论点验证

已证实 (95%) we propose Dynamic Variance-adaptive Advantage Optimization (DVAO). DVAO elegantly bridges the gap between stability and objective synergy by dynamically adjusting the combination weights based on the empirical reward variance of each objective within the rollout group.
The paper provides a complete specification of DVAO in paragraphs 31-33, including the mathematical formulation for dynamic variance-adaptive weights. The method is fully defined and empirically evaluated across multiple benchmarks.
已证实 (95%) we mathematically prove that DVAO not only bounds the advantage magnitude for stable training but also introduces a self-adaptive cross-objective regularization mechanism.
已证实 (90%) we theoretically expose the fundamental flaws of existing scalarization methods in multi-reward GRPO-namely magnitude explosion and objective isolation-and propose Dynamic Variance-adaptive Advantage Optimization to address these limitations.
Proposition 1 (paragraphs 24-26, proof in paragraphs 57-62) mathematically demonstrates magnitude explosion in reward combination. Paragraphs 27-29 analyze objective isolation in advantage combination. DVAO is proposed as the solution.
已证实 (90%) DVAO is a fully dynamic, hyperparameter-free weighting scheme that we mathematically prove maintains bounded advantage magnitudes while introducing an implicit cross-objective regularization mechanism to promote synergistic learning.
The weighting scheme is defined in paragraphs 31-33 as fully data-driven (computed from empirical variance). Proposition 2 proves bounded magnitudes, and Proposition 3 proves the cross-objective regularization mechanism.
已证实 (98%) Proposition 1. For a fixed query x_i, let ρ^i_kl denote the sample correlation between A_k and A_l within the group rollout. The reward combination method and the advantage combination method satisfy: E_j[(A^(i,j)_sum)^2] ≥ E_j[(A^(i,j))^2], with equality if and only if ρ_kl = 1 for all k ≠ l.
This is a formal mathematical proposition with a complete proof provided in paragraphs 57-62. The proof derives the expected squared advantage for both methods and establishes the inequality through correlation analysis.
已证实 (90%) This result reveals that the reward combination method, despite its simplicity, produces advantages with larger squared magnitude on average, leading to larger policy gradients.
已证实 (98%) Proposition 2. For a fixed query x_i and rollout group {y_j}^G_{j=1} ~ π_θ(•|x_i), the reward combination method produces a pointwise larger advantage magnitude than DVAO: |A^(i,j)_sum| ≥ |A^(i,j)_DVAO|, with equality if and only if Cov(r^(i,j)_k, r^(i,j)_l) = σ^i_k σ^i_l for all k ≠ l, i.e., all reward pairs are perfectly positively correlated within the rollout group.
This is a formal mathematical proposition with a complete proof provided in paragraphs 63-73. The proof uses Cauchy-Schwarz inequality to establish the pointwise bound on advantage magnitudes.
已证实 (98%) Proposition 3. For a fixed query x_i, and rollout group {y_j}^G_{j=1} ~ π_θ(•|x_i), the sensitivity of the combined advantage with respect to the k-th raw reward r^(i,j)_k for the advantage combination method and DVAO are respectively given by: ∂A^(i,j)/∂r^(i,j)_k = w_k/σ^i_k and ∂A^(i,j)_DVAO/∂r^(i,j)_k = A^(i,j)_DVAO/σ^i_sum.
This is a formal mathematical proposition with a complete proof provided in paragraphs 74-86. The proof derives the partial derivatives for both methods using standard calculus and the quotient rule.
已证实 (85%) While the sensitivity of A^(i,j) strictly depends on the isolated advantage of the k-th objective, the sensitivity of A^(i,j)_DVAO adaptively depends on the cross-term A^(i,j)_DVAO, allowing it to aggregate global performance information across all objectives within the rollout group.
This is the interpretation of Proposition 3's mathematical result. The sensitivity formula for DVAO (∂A_DVAO/∂r_k = A_DVAO/σ_sum) shows dependency on the combined advantage, while advantage combination's sensitivity depends only on isolated w_k/σ_k.
已证实 (85%) DVAO introduces an implicit crossobjective regularization mechanism. The learning signal for any individual objective is dynamically modulated by the model's global multi-objective performance on that specific rollout.
This claim follows from Proposition 3's sensitivity analysis. The mathematical result shows that DVAO's gradient contribution for objective k is scaled by the overall multi-objective advantage A_DVAO, creating implicit cross-objective regularization.
已证实 (95%) we focus specifically on mathematical reasoning and tool-use tasks to evaluate our proposed DVAO algorithm.
This is a clearly stated design choice in paragraph 46. The paper evaluates on mathematical reasoning (AIME, MATH500, OlympiadBench, AMC23) and tool-use (BFCL-v4) tasks.
已证实 (95%) For mathematical reasoning task, we evaluate models on AIME-2024, AIME-2025, MATH500, OlympiadBench, and AMC23.
The benchmarks are explicitly listed in paragraph 46: AIME-2024, AIME-2025, MATH500, OlympiadBench, and AMC23 for mathematical reasoning.
已证实 (95%) In mathematical reasoning tasks, we focus on two main objectives: accuracy and length constrain. For tool-use task, we focus on two main objectives: tool-use correctness and format compliance.
The objectives are clearly stated in paragraph 46: accuracy and length constraint for mathematical reasoning; tool-use correctness and format compliance for tool-use.
已证实 (95%) we use Qwen3-4B-Base and Qwen3-8B-Base for the mathematical reasoning tasks, and Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct for the tool-use tasks.
The model choices are explicitly stated in paragraph 47: Qwen3-4B-Base and Qwen3-8B-Base for math reasoning; Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct for tool-use.
已证实 (95%) we use DAPO-MATH-17K for mathematical reasoning task, which consists of 17k prompts, each paired with an interger as the answer
The training dataset is specified in paragraph 87: DAPO-MATH-17K with 17k prompts, each paired with an integer answer.
已证实 (95%) we use the accuracy reward r_acc and the length reward r_length for mathematical reasoning task, and use the accuracy reward r_acc and the format reward r_format for tool-use task.
The reward functions are explicitly stated in paragraph 87: accuracy reward and length reward for math; accuracy reward and format reward for tool-use.
已证实 (95%) r_length ∈ {0, 1} checks whether the model's output remains within the target length l, which is set 4,000 tokens for all remaining experiments
The length reward definition and target length are explicitly stated in paragraph 87: r_length ∈ {0, 1} checking output within target length of 4,000 tokens.
已证实 (95%) we utilize the AdamW optimizer with a constant learning rate of 1 × 10^-6.
The optimizer and learning rate are explicitly stated in paragraph 87: AdamW optimizer with constant learning rate of 1 × 10^-6.
已证实 (95%) the prompt batch size is 128 and we sample G = 16 responses for each prompt.
The batch size and rollout sampling are explicitly stated in paragraph 87: prompt batch size 128, G = 16 responses per prompt.
已证实 (95%) we train 500 steps to ensure convergence.
The training duration is explicitly stated in paragraph 87: 500 steps to ensure convergence.

... 共 40 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

局限与证据边界

本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-27T01:27:10+00:00 · 数据来源：Paper Collector