TL;DR
UniVidX leverages Video Diffusion Model priors for unified video generation across 15 tasks using Stochastic Condition Masking, Decoupled Gated LoRA, and Cross-Modal Self-Attention. It achieves state-of-the-art performance with exceptional data efficiency, trained on fewer than 1,000 videos.
26
已证实
12
证据不足
8
无法验证
N/A
可复现性
置信度
79%

核心问题

How can a unified multimodal framework leverage Video Diffusion Model priors to support versatile video generation across multiple input-output paradigms (Text→X, X→X, Text&X→X) without training separate networks for each task?

核心方法

{'approach': 'The framework builds on Wan2.1-T2V-14B backbone with three innovations: Stochastic Condition Masking dynamically partitions modalities into conditions and targets during training, Decoupled Gated LoRA assigns independent gated adapters per modality to prevent interference, and Cross-Modal Self-Attention enables inter-modal interaction through shared key/value mechanisms. Two instantiations are trained: UniVid-Intrinsic (924 synthetic videos) and UniVid-Alpha (484 videos).', 'key_components': ['UniVidX is designed as a unified framework leveraging VDM priors for versatile multimodal generation.', 'The methodology comprises four main components addressing different aspects of the unified generation problem.', 'Two instantiations (UniVid-Intrinsic and UniVid-Alpha) will be detailed with their training configurations.', 'UniVid-Intrinsic processes RGB, albedo, irradiance, and normal maps across three paradigms.', 'Roughness, metallic, and depth maps are excluded from UniVid-Intrinsic due to annotation scarcity and redundancy with normals.', 'UniVid-Alpha processes blended RGB, alpha matte, foreground, and background layers.', 'Alpha is replicated across three channels for VAE encoder compatibility.', 'The background layer is trained to automatically inpaint regions occluded by the foreground.'], 'section_ids': ['sec_3', 'sec_8']}

论点验证

已证实 (95%) We present UniVidX. It is a unified multimodal framework designed to leverage VDM priors for versatile video generation, which incorporates three key designs: 1) Stochastic Condition Masking (SCM)... 2) Decoupled Gated LoRA (DGL)... and 3) Cross-Modal Self-Attention (CMSA).
The paper provides complete specification of all three key designs (SCM in Sec 3.1, DGL in Sec 3.2, CMSA in Sec 3.3) with mathematical formulations, and validates them through comprehensive experiments and ablation studies.
已证实 (95%) We instantiate UniVidX in two multimodal domains: 1) UniVid-Intrinsic, which models among RGB videos and the corresponding intrinsic maps (albedo/irradiance/normal), and 2) UniVid-Alpha, which processes blended RGB (BL), alpha matte (Alpha), foreground (FG), and background (BG) layers.
Both instantiations are clearly specified in Section 3.4 with explicit modality definitions: UniVid-Intrinsic processes RGB, albedo, irradiance, normal; UniVid-Alpha processes BL, Alpha, FG, BG.
已证实 (85%) Both models demonstrate versatility, supporting three paradigms (Text→X; X→X; Text&X→X) and collectively covering 15 distinct tasks.
The three paradigms (Text→X, X→X, Text&X→X) are clearly defined and demonstrated across multiple tasks. The paper shows examples for each paradigm, though the full list of 15 tasks is in the appendix.
证据不足 (60%) Both models demonstrate exceptional data efficiency. They exhibit robust generalization to out-of-distribution, in-the-wild scenarios, despite being trained on limited domain-specific datasets.
Training data sizes are specified (900 and 484 videos), and qualitative OOD examples are shown (animals in p_40). However, there's no systematic quantitative evaluation of generalization across diverse out-of-distribution scenarios - only isolated qu
证据不足 (55%) Both demonstrate state-of-the-art performance across diverse tasks and robust in-the-wild generalization, despite using limited training data (<1k videos).
The claim of 'state-of-the-art across diverse tasks' is overstated. While video matting shows SOTA (MAD 4.24), normal estimation on Sintel (MAE 15.73°) is competitive but not SOTA (Stable Normal: 14.69°). The comparison is not comprehensive across al
证据不足 (50%) We propose Stochastic Condition Masking (SCM), a strategy that unifies diverse video tasks into one diffusion model. Specifically, SCM is built upon a T2V backbone, selected for two strategic reasons: (i) it inherently possesses the capability to process pure text inputs, and (ii) its latent space is adaptable, allowing us to seamlessly incorporate visual inputs alongside text.
The rationale for choosing T2V backbone is stated but not empirically validated. No ablation compares T2V vs V2V or other backbone choices to justify this design decision.
已证实 (95%) During training, we employ a dynamic random partitioning strategy that splits Z into two mutually exclusive subsets: 1) Target Subset Z tgt... 2) Condition Subset Z cond...
The dynamic random partitioning strategy is clearly specified with mathematical formulation for target and condition subsets.
已证实 (95%) We implement this logical partition via timestep manipulation. Specifically, for the target subset Z tgt, we denote the clean latents as x T. The intermediate noisy state z T t is obtained via linear interpolation between the Gaussian noise ε ∼ N(0, I) and the clean data x T at timestep t ∈ [0, 1]; the latents in Z cond are fixed at t = 1, denoted as z C 1, serving as unnoised conditions.
The timestep manipulation implementation is clearly specified with mathematical formulation including the flow matching objective in Equation 1.
已证实 (90%) DGL assigns independent LoRAs to each specific modality. Crucially, these LoRAs are activated only when their corresponding modality serves as a generation target.
The DGL design is clearly specified and validated through ablation study (p_52-54) showing clear modality disentanglement vs chaotic attention maps in the shared-parameter variant.
已证实 (95%) For the k-th modality, we introduce a specific parameter update ΔW k = B k A k, where B k ∈ R d×r and A k ∈ R r×d are learnable low-rank matrices (r ≪ d). This design decouples the processing capabilities for different modalities into distinct parameter spaces, isolating disparate data distributions.
Clear mathematical formulation provided for the LoRA parameter update with rank specification.
已证实 (85%) When the k-th modality serves as a generation target (noisy input), the gate is activated (m k = 1); when it serves as a condition (clean input), the gate is suppressed (m k = 0), which bypasses the adapter, maximizing the utilization of the VDM's native encoding capability to extract robust semantic features from the visual context without domain-shift interference.
The gating mechanism is clearly specified and validated through ablation showing quantitative impact (albedo PSNR drops 1.87 dB without gating).
已证实 (85%) We introduce Cross-Modal Self-Attention (CMSA) to accelerate interaction and fusion across modalities. Specifically, we aggregate the keys and values from all modalities to form a shared context, while keeping the queries modality-specific.
CMSA is clearly specified with mathematical formulation and validated through ablation showing improved cross-modal consistency vs vanilla attention.
证据不足 (40%) We deliberately exclude [roughness and metallic maps] from our target modalities. This decision is driven by two factors. First, reliable ground-truth annotations for material properties are scarce and difficult to curate... Second, we leverage the robust priors of pre-trained VDMs. We observe that the VDM possesses an inherent capacity to infer material properties from context.
The rationale for excluding roughness/metallic maps is stated but not empirically validated. No experiments demonstrate that VDM can infer material properties or compare performance with/without these modalities.
证据不足 (40%) We also exclude depth maps from our formulation. Depth is primarily a macro-geometric attribute rather than a direct photometric component of the shading equation. Moreover, our framework already incorporates surface normals, which capture the finer local geometric details essential for shading computation.
The rationale for excluding depth maps is stated but not empirically validated. No ablation compares depth vs normal inclusion or demonstrates that normals alone are sufficient.
已证实 (95%) To ensure compatibility, we adapt the inherently single-channel Alpha by replicating it across three channels before feeding it into the VAE. This allows us to process alpha matte within the same latent space as color (RGB).
Clear specification of the alpha channel replication approach for VAE compatibility.
已证实 (95%) We build our framework upon the Wan2.1-T2V-14B backbone. The rank of LoRA modules in DGL is set to 32 for all modalities, resulting in a total of 385M trainable parameters.
Concrete implementation details specified: Wan2.1-T2V-14B backbone, LoRA rank 32, 385M trainable parameters.
已证实 (95%) We employ a unified optimization strategy for both UniVid-Intrinsic and UniVid-Alpha, using AdamW (β1 = 0.9, β2 = 0.999, weight decay=10-4) coupled with a Cosine Annealing scheduler that decays the learning rate from an initial 1 × 10-4 to 1 × 10-6.
Complete optimization hyperparameters specified: AdamW with β1=0.9, β2=0.999, weight decay=10^-4, cosine annealing from 1e-4 to 1e-6.
已证实 (95%) Training is conducted on 4× NVIDIA H100 GPUs, utilizing BFloat16 (BF16) mixed precision to maximize throughput. Moreover, both models process video clips of 21 frames, with a per-GPU batch size of 1. Under this setup, UniVid-Intrinsic is trained for 6,000 steps, while UniVid-Alpha is trained for 5,000 steps.
Complete training setup specified: 4× H100 GPUs, BF16, 21 frames, batch size 1, 6000/5000 steps.
已证实 (95%) We construct a synthetic dataset InteriorVid. It comprises 924 high-quality indoor video clips, each consisting of 21 frames at a resolution of 480 × 640, with paired ground-truth for albedo, irradiance, and normal maps.
Complete dataset specification: 924 clips, 21 frames, 480×640 resolution, paired ground-truth for albedo/irradiance/normal.
已证实 (95%) For UniVid-Alpha, we utilize VideoMatte240K, a widely adopted dataset for video matting featuring human foregrounds with paired ground-truth alpha mattes. We use 484 videos from this dataset to train our model, with resolution resized to 432 × 768.
Complete dataset specification: 484 videos from VideoMatte240K, 432×768 resolution.

... 共 46 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

局限性（作者自述）

本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-04T07:11:08+00:00 · 数据来源：Paper Collector