Flow-OPD integrates On-Policy Distillation into Flow Matching models, replacing sparse rewards with dense trajectory-level supervision from specialized teachers.
论文原始摘要
Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a 'seesaw effect' of competing metrics and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in the large language model community, we propose Flow-OPD, the first unified post-training framework that integrates on-policy distillation into Flow Matching models. Flow-OPD adopts a two-stage alignment strategy: it first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; it then establishes a robust initial policy through a Flow-based Cold-Start scheme and seamlessly consolidates heterogeneous expertise into a single student via a three-step orchestration of on-policy sampling, task-routing labeling, and dense trajectory-level supervision. We further introduce Manifold Anchor Regularization (MAR), which leverages a task-agnostic teacher to provide full-data supervision that anchors generation to a high-quality manifold, effectively mitigating the aesthetic degradation commonly observed in purely RL-driven alignment. Built upon Stable Diffusion 3.5 Medium, Flow-OPD raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference alignment and exhibiting an emergent 'teacher-surpassing' effect. These results establish Flow-OPD as a scalable alignment paradigm for building generalist text-to-image models. The codes and weights will be released in: https://github.com/CostaliyA/Flow-OPD .
Paper Collector 中文速览
首个将在线策略蒸馏引入Flow Matching的统一后训练框架
方法概述
采用两阶段策略：先通过单奖励GRPO微调培养领域专家教师模型；再通过Flow冷启动建立初始策略，通过在线策略采样、任务路由标注和密集轨迹级监督三步编排，将异质知识整合到单一学生模型中；引入MAR利用任务无关教师提供全数据监督锚定高质量流形
核心贡献
提出Flow-OPD两阶段对齐框架，集成MAR流形锚点正则化，有效解决多任务对齐中的梯度干扰和奖励稀疏问题，实现GenEval 63→92、OCR 59→94的显著提升
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
How can On-Policy Distillation be effectively integrated into Flow Matching models to resolve the multi-task degradation problem caused by reward sparsity and gradient interference in GRPO-based methods?
核心方法
Flow-OPD employs a three-stage framework: training domain-expert teachers via Flow-GRPO, cold-start initialization through SFT or model merging, and multi-teacher online distillation with dynamic task routing. The approach converts deterministic ODEs to stochastic SDEs for exploration and uses hard routing to activate single domain experts, with Manifold Anchor Regularization preserving aesthetic quality during functional alignment.
方法组件
- Flow-OPD reformulates multi-task alignment via dense supervision on self-generated trajectories rather than sparse scalar rewards.
- Domain-expert teachers are first trained using Flow-GRPO to ensure each expert reaches its performance ceiling in isolation.
- Multi-Teacher Online Distillation dynamically routes online samples to specific teachers for fine-grained, domain-specific guidance.
- Manifold Anchor Regularization preserves aesthetic integrity by decoupling functional alignment from generative prior collapse.
- Qwenvl Score uses adapted prompt from Flow-GRPO framework.
- Evaluation model is Qwen3-30B-A3B-Instruct-2507.
论点验证
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available
- No training/evaluation data available
- Missing all training hyperparameters (learning rate, batch size, epochs/iterations, optimizer settings)
- Random seeds not specified
- Base model architecture not clearly specified (only references Flow-GRPO)
- Dynamic routing mechanism implementation details not provided
- Manifold Anchor Regularization implementation details and parameters missing
- Cold-start initialization procedure not described in detail
- Training data splits only referenced (Flow-GRPO splits) but not provided
- Preprocessing steps not documented
局限与证据边界
- A primary constraint is the performance ceiling imposed by teacher.
- The specialized teachers may harbor latent biases present in their training data, which could be inadvertently distilled into the student model.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-26T13:22:11+00:00 · 数据来源：Paper Collector