LPM 1.0: Video-based Character Performance Model

核心问题

How can a video-generative system for single-person conversational performance simultaneously achieve expressive quality, real-time inference, and long-horizon stability—the "performance trilemma"?

核心方法

{'approach': 'The authors develop a full-stack framework comprising a curated multimodal dataset with aggressive filtering, Base LPM (17B bidirectional Diffusion Transformer with interleaved dual-audio injection), and Online LPM (distilled causal backbone-refiner architecture). A four-stage distillation curriculum converts the offline bidirectional model into a real-time streaming generator, evaluated on the newly introduced LPM-Bench benchmark with 1,000 test cases.', 'key_components': ['GVHMR estimates the facing direction of the human body using the SMPL 3D body model, while SLAM estimates camera pose.', 'Frames are classified into four viewpoint categories based on the angle between camera orientation and human facing direction.', 'Representative frames from each category provide the model with direct visual evidence for viewpoint-dependent appearance.', 'The model is built on a DiT architecture with three-stage blocks: self-attention with AdaLN, multi-modal cross-attention, and FFN with AdaLN.', 'An interleaved dual-audio injection strategy alternately processes speak and listen audio on even and odd transformer layers.', 'Multi-identity image token injection concatenates patchified reference images into the self-attention sequence for identity preservation.', 'The model jointly learns speech-driven temporal dynamics, content-aware listening reactions, text-conditioned control, and identity preservation.', 'The model initializes from Wan2.1-I2V (16B) and trains audio pathways progressively: speaking first, then listening, then combined conversation data.', 'Value-projection weights in audio cross-attention layers are zero-initialized for stable adaptation during audio injection.', 'Original CLIP image cross-attention blocks and channel-mask-image in self-attention are removed for simplification.'], 'section_ids': ['sec_10', 'sec_13', 'sec_15', 'sec_17', 'sec_20', 'sec_21', 'sec_23', 'sec_33', 'sec_35']}

论点验证

... 共 57 个论点

可复现性评估

较低可复现性 (0%)

局限性（作者自述）

• On Domain 2, a notable asymmetry emerges: speak recall reaches 94.05% but listen recall drops to 81.05%. This gap confirms the challenge described in Section 2.2: in domains with higher acoustic variability, ambient noise and off-screen speech cause a fraction of true listening frames to be misclassified as speaking.
• A residual weakness is that listen_dialogue clips are over-predicted as conversation (25.4% after fine-tuning versus 19.1% for Gemini), suggesting that further calibration on this boundary case could yield additional gains.
• Despite strong overall generation quality, the base model exhibits two notable limitations in conversational scenarios. First, in speaking mode, large or rapid motions can introduce visual artifacts-most notably hand and limb distortions (e.g., unnatural bending, missing fingers, or implausible joint angles), physically inconsistent body configurations, and occasional degradation in lip-sync accuracy. Second, in listening mode, the model occasionally produces overly static outputs, generating near-frozen frames that lack the natural micro-movements and non-verbal responses expected of an attentive listener.
• In practice, we observe that DMD alone may lead to mode collapse.