TL;DR
This paper diagnoses three pathologies in DiT residual connections—magnitude inflation, gradient decay, and redundancy—and proposes DAR, a learnable timestep-adaptive routing mechanism. DAR achieves 8.75× faster training and 2.11 FID improvement on SiT-XL/2.

论文原始摘要

Diffusion Transformers (DiTs) have become a de facto backbone of modern visual generation, and nearly every major axis of their design -- tokenization, attention, conditioning, objectives, and latent autoencoders -- has been extensively revisited. The residual stream that governs how information accumulates across layers, however, has been directly inherited from the original Transformer. In this paper, we present a systematic empirical analysis of cross-layer information flow in DiTs, jointly along depth and denoising timestep, and identify three concrete symptoms of traditional residual addition, namely monotonic forward magnitude inflation, sharp backward gradient decay, and pronounced block-wise redundancy. Motivated by this diagnosis, we propose Diffusion-Adaptive Routing (\textsc{DAR}), a drop-in residual replacement that performs \emph{learnable, timestep-adaptive, and non-incremental} aggregation over the history of sublayer outputs. Moreover, the proposed \textsc{DAR} is compatible with many modern Transformer enhancement methods, such as REPA. On ImageNet $256\times256$, \textsc{DAR} improves SiT-XL/2 by $2.11$ FID ($7.56$ vs.\ $9.67$) and matches the baseline's converged quality with $8.75\times$ fewer training iterations. Stacked on top of REPA, it yields a $2\times$ training acceleration in the early stage, suggesting cross-layer information routing as an underexplored design axis in diffusion modeling, one that operates orthogonally to existing representation-alignment objectives. Beyond pretraining, \textsc{DAR} can also be applied during the fine-tuning stage of large-scale T2I models and preserves high-frequency details during Distribution Matching Distillation.

Paper Collector 中文速览

提出DAR改进DiT跨层信息路由，显著提升生成质量并加速训练

方法概述

系统分析DiT跨层信息流，识别前向幅度膨胀、后向梯度衰减和块冗余问题。设计DAR替代残差加法，对历史子层输出执行可学习、时间步自适应、非增量聚合。兼容REPA等方法，可应用于预训练和微调阶段。

核心贡献

发现传统残差连接三大问题，提出可学习、时间步自适应的DAR聚合机制，提升SiT-XL/2的FID达2.11，训练加速8.75倍

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

30
已证实
3
证据不足
3
无法验证
N/A
可复现性
置信度
85%

核心问题

What are the pathologies of standard pre-normalized residual connections in Diffusion Transformers, and can learnable, timestep-adaptive cross-layer routing improve training efficiency and generation quality?

核心方法

The authors diagnose three symptoms in standard DiT residual streams through empirical analysis of forward magnitude, backward gradients, and block similarity across depth and denoising timesteps. They propose DAR, which replaces fixed residual addition with softmax-weighted aggregation over preceding sublayer outputs using adaLN-modulated queries for timestep awareness. Chunked aggregation reduces memory complexity from O(Ld) to O((S+N)d), with a fused Triton kernel implementation for efficiency.

方法组件

论点验证

已证实 (85%) We conduct, to the best of our knowledge, the first comprehensive investigation of the cross-layer information flow in DiTs along both depth and denoising timestep and identify three concrete symptoms of the prevailing residual structure in DiTs, that is, forward magnitude inflation, backward gradient decay, and block-wise redundancy.
The paper provides quantitative evidence for all three symptoms in Section 3 and Fig. 2: forward magnitude inflation (15.5 to 1576, ~100×), backward gradient decay (early blocks ~5×10^-7, later blocks near zero), and block-wise redundancy (cosine sim
已证实 (90%) We propose DAR, a drop-in residual replacement for DiTs that performs learnable, timestep-adaptive, and non-incremental aggregation.
DAR is fully specified in Section 4 with equations (Eq. 5-7), implementation details, query parameterizations, chunked aggregation design, and experimental validation. The method performs softmax-weighted aggregation (learnable), can be timestep-cond
已证实 (85%) On SiT, we achieve 8.75× faster training and a 2.11 FID improvement over the baseline.
The paper provides specific FID numbers: baseline achieves 9.03 FID at 400K iterations, DAR achieves 6.92 FID (2.11 improvement). For training speed, DAR at 400K iterations matches baseline at 3.5M iterations, which is 8.75× faster. These are concret
已证实 (85%) Stacked on top of REPA [66], it yields a 2× training acceleration in the early stage over REPA alone.
Table 3 shows DAR+REPA achieves FID 7.09 at 100K iterations while REPA alone achieves FID 9.89 at 100K and 6.89 at 200K. DAR+REPA at 100K surpasses REPA at 200K, indicating approximately 2× early-stage acceleration.
已证实 (95%) The forward hidden-state magnitude grows monotonically from ∼ 15.5 at block 1 to ∼ 1576 at block 28, corresponding to roughly 100× inflation.
Direct quote from p_16 with specific numbers: 'The forward hidden-state magnitude grows monotonically from ~15.5 at block 1 to ~1576 at block 28, corresponding to roughly 100× inflation.' This is a precise quantitative measurement from Fig. 2.
已证实 (90%) Early blocks receive substantial signal (∼ 5 × 10^-7), whereas later blocks are lower by more than an order of magnitude and remain close to zero throughout the deep stack.
Direct quote from p_17: 'Early blocks receive substantial signal (~5×10^-7), whereas later blocks are lower by more than an order of magnitude and remain close to zero throughout the deep stack.' Specific quantitative measurement.
已证实 (90%) The per-token cosine similarity between consecutive block outputs stays above 0.9 throughout the deep stack, indicating that neighboring deep blocks produce highly similar representations.
Direct quote from p_17: 'The per-token cosine similarity between consecutive block outputs stays above 0.9 throughout the deep stack.' Specific quantitative measurement from Fig. 2.
已证实 (75%) Although the baseline never sees a router during training, its counterfactual importance map already varies systematically along t at both depths, with the preferred sources at high noise differing visibly from those at low noise.
The paper describes the counterfactual importance analysis in p_18 and states that 'its counterfactual importance map already varies systematically along t at both depths, with the preferred sources at high noise differing visibly from those at low n
已证实 (75%) DAR's learned weights provide the missing degree of freedom suggested by this diagnostic: the softmax concentrates sharply on a small subset of historical sources, and this selection itself shifts smoothly with t at both shallow and deep blocks.
The paper states in p_18 that 'DAR's learned weights provide the missing degree of freedom' and that 'the softmax concentrates sharply on a small subset of historical sources, and this selection itself shifts smoothly with t.' Fig. 3 visualizes this.
已证实 (95%) We analyze two models: a vanilla SiT-XL/2 baseline and a static variant of DAR with chunk size S = 4.
Direct statement in p_14: 'We analyze two models: a vanilla SiT-XL/2 baseline and a static variant of DAR with chunk size S = 4.' This is a clear methodological design choice.
已证实 (95%) Both models are checkpointed after 600K training iterations, and diagnostics are computed on 4096 ImageNet samples.
Direct statement in p_14: 'Both models are checkpointed after 600K training iterations, and diagnostics are computed on 4096 ImageNet samples.' Clear methodological specification.
已证实 (95%) The proposed DAR replaces the unweighted sum with a softmax-weighted aggregation.
Equation in p_28 shows the softmax-weighted aggregation formula: h_l = Σ α_{i→l}(t) v_i where α is computed via softmax. This is the core DAR mechanism.
已证实 (90%) The final layer of the t-embedder is zero-initialized, so that e(t) = 0 at initialization and the model exactly recovers the pure static variant at the start of training.
Direct statement in p_32: 'The final layer of the t-embedder is zero-initialized, so that e(t) = 0 at initialization and the model exactly recovers the pure static variant at the start of training.' Clear design specification.
已证实 (90%) Each chunk n is summarized by a single representation c_n := v_nS, i.e., the output of its last sublayer.
Statement in p_33: 'Each chunk n is summarized by a single representation c_n := v_nS, i.e., the output of its last sublayer.' Clear design specification.
已证实 (80%) Both timestep-aware variants substantially outperform the timestep-blind baseline at matched compute.
The paper states in p_35: 'both timestep-aware variants substantially outperform the timestep-blind baseline at matched compute' and references Table 2. However, the actual FID numbers from Table 2 are not provided in the text.
已证实 (85%) Both the attention and MLP aggregator inputs sit well above the raw input latents x_t baseline (R^2 ≈ 0.80) at every depth, exceed 0.95 within the first five blocks, and remain close to 1.0 throughout the deep stack.
Direct quote from p_36 with specific R² values: 'Both the attention and MLP aggregator inputs sit well above the raw input latents x_t baseline (R² ≈ 0.80) at every depth, exceed 0.95 within the first five blocks, and remain close to 1.0 throughout t
已证实 (90%) Stacking DAR on top of REPA improves FID from 9.89 to 7.09 at 100K iterations and from 6.89 to 5.92 at 200K iterations.
Direct quote from p_37: 'stacking DAR on top of REPA improves FID from 9.89 to 7.09 at 100K iterations and from 6.89 to 5.92 at 200K iterations.' Specific quantitative results from Table 3.
已证实 (90%) DAR+REPA at 100K already surpasses the 200K FID of REPA alone, indicating that the routing-level and representation-level accelerations compound rather than offset each other.
Direct quote from p_37: 'DAR+REPA at 100K already surpasses the 200K FID of REPA alone, indicating that the routing-level and representation-level accelerations compound rather than offset each other.' The numbers (7.09 at 100K vs 6.89 at 200K) suppo
已证实 (90%) Proposition 1 (U-shaped cost of chunked aggregation): Let L > 0 and α ∈ (0, 1). Then L(S) is strictly decreasing on (0, S^⋆) and strictly increasing on (S^⋆, ∞), where S^⋆ = √(αL/(1-α)). Consequently, L(S) is U-shaped and has a unique global minimizer at S^⋆.
Proposition 1 is stated in p_40-41 with the formula S^⋆ = √(αL/(1-α)). A complete proof is provided in p_48-54 (Appendix B), showing the derivative analysis and proving the U-shaped property.
已证实 (85%) For SiT-XL/2 (depth 28, two sublayers per chunk, hence L = 56), Eq. (9) predicts S^⋆ ∈ [3.7, 4.9] over the realistic range α ∈ [0.4, 0.7], identifying S = 4 as the model-predicted optimum.
Direct quote from p_55: 'For SiT-XL/2 (depth 28, two sublayers per chunk, hence L = 56), Eq. (9) predicts S^⋆ ∈ [3.7, 4.9] over the realistic range α ∈ [0.4, 0.7], identifying S = 4 as the model-predicted optimum.' Table 4 shows S=4 performs best.

... 共 36 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

局限与证据边界

本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-26T01:26:55+00:00 · 数据来源：Paper Collector