TL;DR
CoInteract synthesizes physically-consistent human-object interaction videos using a Diffusion Transformer with embedded structural priors. It introduces Human-Aware Mixture-of-Experts for hand/face quality and dual-stream co-generation jointly training RGB and HOI structure streams.
CoInteract synthesizes physically-consistent human-object interaction videos using a Diffusion Transformer with embedded structural priors. It introduces Human-Aware Mixture-of-Experts for hand/face quality and dual-stream co-generation jointly training RGB and HOI structure streams.
46
已证实
4
证据不足
1
无法验证
N/A
可复现性
核心问题
The paper investigates how to synthesize physically-consistent human-object interaction videos that maintain structural stability and physical plausibility, addressing limitations of RGB-centric diffusion models lacking 3D spatial understanding.
核心方法
{'approach': 'The framework introduces a Human-Aware Mixture-of-Experts module that routes tokens to region-specialized experts for hands and faces using spatially-supervised routing. It proposes a dual-stream co-generation paradigm that jointly trains an RGB stream with an auxiliary HOI structure stream within a shared DiT backbone, using asymmetric co-attention to enable zero-overhead inference.', 'key_components': ['Video diffusion models have evolved to use DiT-style backbones for modeling spatiotemporal tokens with global attention.', 'RGB-centric models remain fragile in HOI scenarios with weak constraints on contact geometry and body topology.', 'Common failure modes include hand/face distortions and contact violations such as interpenetration.', 'Recent multi-stream co-generation methods target general video synthesis but not HOI-specific challenges.', 'CoInteract injects interaction-structure supervision and region-specific specialization into a shared DiT backbone.', 'CoInteract is an end-to-end framework for speech-driven HOI video synthesis using dual reference images and motion frames.', 'The framework synthesizes HOI videos that are structurally stable and physically plausible.', 'Unlike conventional video diffusion models, CoInteract explicitly injects interaction structure into a shared DiT backbone.'], 'section_ids': ['sec_3', 'sec_6']}
论点验证
已证实 (85%)
We present CoInteract, an end-to-end framework that introduces a Spatially-Structured Co-Generation paradigm for physically-consistent HOI video synthesis.
The paper provides detailed architecture description (Section 3), quantitative results in Table 1 showing improvements across multiple metrics, qualitative results in Figure 5, and ablation studies in Table 2. The framework is fully specified and exp
The paper provides detailed architecture description (Section 3), quantitative results in Table 1 showing improvements across multiple metrics, qualitative results in Figure 5, and ablation studies in Table 2. The framework is fully specified and exp
已证实 (80%)
We propose a Human-Aware Mixture-of-Experts (MoE) to improve hand and face quality. Using face and hand bounding boxes as supervision, a lightweight router learns to dispatch tokens to region-specialized experts that enhance structural fidelity with only a marginal increase in parameters.
The MoE architecture is fully described (p_28-31), and Table 2 ablation shows removing MoE causes drops in HQ (0.724→0.712) and FaceSim (0.696→0.682). The 'marginal increase in parameters' claim is quantified as 1.04× overhead in Table 2.
The MoE architecture is fully described (p_28-31), and Table 2 ablation shows removing MoE causes drops in HQ (0.724→0.712) and FaceSim (0.696→0.682). The 'marginal increase in parameters' claim is quantified as 1.04× overhead in Table 2.
已证实 (85%)
We introduce a dual-stream co-generation paradigm to enforce physical plausibility. An auxiliary HOI structure stream—where the human body is reduced to a silhouette while the object retains its RGB appearance—is jointly trained with the RGB stream within a shared DiT backbone.
The dual-stream paradigm is fully described in p_16, and Table 2 ablation provides strong evidence: removing HOI stream causes VLM-QA to drop from 0.72 to 0.48 (-33.3%), demonstrating its importance for physical plausibility.
The dual-stream paradigm is fully described in p_16, and Table 2 ablation provides strong evidence: removing HOI stream causes VLM-QA to drop from 0.72 to 0.48 (-33.3%), demonstrating its importance for physical plausibility.
已证实 (75%)
We propose CoInteract, a novel end-to-end framework for speech-driven HOI video synthesis that embeds human structural priors and interaction geometry constraints directly into the DiT backbone, ensuring both physical plausibility and structural consistency without relying on external preprocessing or post-processing.
The framework is described in detail and validated experimentally. The claim about 'no external preprocessing or post-processing' refers to inference time, which is supported by the architecture where the HOI branch is removed at inference. However,
The framework is described in detail and validated experimentally. The claim about 'no external preprocessing or post-processing' refers to inference time, which is supported by the architecture where the HOI branch is removed at inference. However,
已证实 (80%)
We introduce a Human-Aware Mixture-of-Experts (MoE) that uses spatial supervision to guide lightweight specialized experts for hands and faces. This targeted processing ensures high structural fidelity and effectively reduces the artifacts commonly seen in these critical regions, with minimal additional parameters.
Same as claim_2 - MoE is described in detail and validated through ablation. The 'minimal additional parameters' is quantified as 1.04× overhead in Table 2.
Same as claim_2 - MoE is described in detail and validated through ablation. The 'minimal additional parameters' is quantified as 1.04× overhead in Table 2.
已证实 (85%)
We develop a Spatially-Structured Co-Generation paradigm using an asymmetric co-attention mask to embed physical interaction rules into the DiT. This approach forces the model to respect geometric constraints and substantially reduces hand-object interpenetration, while ensuring zero additional computational cost at inference.
The asymmetric co-attention mechanism is described in p_26-27. Table 2 provides strong evidence: full model achieves VLM-QA 0.72 at 1.00× cost, while retaining HOI branch achieves 0.76 at 4.13× cost. The asymmetric strategy enables zero-overhead infe
The asymmetric co-attention mechanism is described in p_26-27. Table 2 provides strong evidence: full model achieves VLM-QA 0.72 at 1.00× cost, while retaining HOI branch achieves 0.76 at 4.13× cost. The asymmetric strategy enables zero-overhead infe
已证实 (80%)
CoInteract explicitly injects interaction structure and body-level consistency into a shared Diffusion Transformer (DiT) backbone.
Design choice fully specified in Section 3 with detailed architecture description. The shared DiT backbone with dual streams and MoE is clearly described.
Design choice fully specified in Section 3 with detailed architecture description. The shared DiT backbone with dual streams and MoE is clearly described.
已证实 (85%)
We introduce a unified co-generation paradigm in which an RGB appearance stream z_r and an auxiliary HOI structure stream z_h are jointly trained within a single DiT backbone.
Design choice fully specified in p_16 with clear description of RGB stream z_r and HOI structure stream z_h jointly trained in single DiT backbone.
Design choice fully specified in p_16 with clear description of RGB stream z_r and HOI structure stream z_h jointly trained in single DiT backbone.
已证实 (80%)
We construct an auxiliary HOI structure stream as a silhouette-like 3-channel rendering obtained by projecting the recovered human mesh to the image plane and fusing the projected object mask. This produces a pixel-aligned structural target that highlights interaction boundaries while discarding RGB texture.
Design choice fully specified in p_16 and p_32, describing how human mesh projection and object mask fusion create the HOI structure stream.
Design choice fully specified in p_16 and p_32, describing how human mesh projection and object mask fusion create the HOI structure stream.
已证实 (85%)
The two streams are tokenized by modality-specific patch embedding layers (with the same patch size) and then fed into shared DiT blocks.
Design choice clearly specified in p_17 - modality-specific patch embedding layers with same patch size feeding into shared DiT blocks.
Design choice clearly specified in p_17 - modality-specific patch embedding layers with same patch size feeding into shared DiT blocks.
已证实 (80%)
Within each DiT block, the two streams share all transformer parameters but employ stream-specific modulation parameters (scale and shift in adaptive layer normalization), enabling a single backbone to specialize feature statistics for appearance versus structure without duplicating the full model.
Design choice specified in p_17 - shared transformer parameters with stream-specific modulation (scale and shift in adaptive layer normalization).
Design choice specified in p_17 - shared transformer parameters with stream-specific modulation (scale and shift in adaptive layer normalization).
已证实 (85%)
We optimize the model with a joint flow-matching objective supervising both streams.
Design choice specified with mathematical formulation in p_18-20, showing joint flow-matching objective for both streams.
Design choice specified with mathematical formulation in p_18-20, showing joint flow-matching objective for both streams.
已证实 (90%)
We set λ_h = 1 unless otherwise stated.
Simple hyperparameter setting stated clearly in p_20.
Simple hyperparameter setting stated clearly in p_20.
已证实 (85%)
We assign each token a 3D coordinate (h, w, t) encoded by 3D Rotary Positional Encoding (3D RoPE).
Design choice specified in p_21 with 3D RoPE encoding for token coordinates.
Design choice specified in p_21 with 3D RoPE encoding for token coordinates.
已证实 (85%)
To preserve pixel-level correspondence between RGB and HOI, we concatenate the two streams along the width dimension, assigning distinct horizontal coordinates—e.g. w ∈ [0, W] for RGB and w ∈ [-W, 0] for HOI—while sharing identical height and time indices.
Design choice specified in p_22 with clear coordinate assignment scheme for dual streams.
Design choice specified in p_22 with clear coordinate assignment scheme for dual streams.
已证实 (85%)
Past motion frames are assigned negative temporal indices (e.g., t ∈ {-N, . . . , -1}), placing them logically before the current generation window to encourage causal motion continuity.
Design choice specified in p_23 with clear temporal indexing scheme for motion frames.
Design choice specified in p_23 with clear temporal indexing scheme for motion frames.
已证实 (85%)
Static reference images are mapped to a far-field temporal location (e.g., t = 30, 31) with a significant offset, encouraging the model to treat them as global identity anchors rather than adjacent frames.
Design choice specified in p_23 with specific temporal locations (t=30, 31) for reference images.
Design choice specified in p_23 with specific temporal locations (t=30, 31) for reference images.
已证实 (85%)
We adopt a two-stage training strategy with an Asymmetric Co-Attention mechanism.
Design choice specified in p_26 with two-stage training and asymmetric co-attention mechanism.
Design choice specified in p_26 with two-stage training and asymmetric co-attention mechanism.
已证实 (80%)
In Stage 1, standard bidirectional attention is applied across both streams for rapid convergence, allowing the model to learn global dependencies between appearance and interaction structure.
Design choice specified in p_26 describing Stage 1 with bidirectional attention.
Design choice specified in p_26 describing Stage 1 with bidirectional attention.
已证实 (85%)
In Stage 2, we enforce an asymmetric attention mask. Under this mask, RGB queries attend only to RGB tokens, making the RGB pathway independent of the HOI branch and thus removable at inference with zero overhead.
Design choice specified in p_26-27 with asymmetric attention mask formulation, enabling RGB independence from HOI branch.
Design choice specified in p_26-27 with asymmetric attention mask formulation, enabling RGB independence from HOI branch.
... 共 51 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- Model architecture specifications (number of layers, hidden dimensions, attention configuration for DiT backbone)
- Training hyperparameters (learning rate, batch size, epochs, optimizer, learning rate schedule)
- Diffusion-specific parameters (number of diffusion steps, noise schedule, guidance scale, sampling method)
- Dataset information (training datasets used, dataset size, train/val/test splits, data preprocessing steps)
- Loss functions and their respective weights for multi-stream co-generation
- Hardware specifications (GPU type, memory requirements, training/inference time)
- Random seeds for reproducibility
- Evaluation metrics and their implementation details
- Baseline methods and comparison protocols
- Input/output specifications (resolution, frame rate, motion frame format)
局限性（作者自述）
- CoInteract instead faithfully preserves the reference scene, which trades marginal aesthetic scores for stronger consistency.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-04-23T07:30:39+00:00 · 数据来源：Paper Collector