WBENCH introduces a multi-turn benchmark evaluating interactive world models across five dimensions (rendering, initialization, interaction, memory, physics) with 289 cases.
论文原始摘要
Interactive world models are advancing rapidly, yet existing benchmarks cover only part of the required competencies, leaving no unified standard for systematic evaluation. To fill this gap, we introduce WBench, a comprehensive multi-turn benchmark for interactive world model evaluation along five dimensions, namely video quality, setting adherence, interaction adherence, consistency, and physics compliance. WBench contains 289 test cases and 1,058 interaction turns, where each case specifies a world setting and a multi-turn interaction sequence, covering diverse scenes, styles, subjects, and both first- and third-person perspectives, together with four interaction types, including navigation, subject action, event editing, and perspective switching. For navigation, WBench unifies text, 6-DoF pose, and discrete-action control, enabling evaluation of models with different native input interfaces. Evaluation uses 22 automatic sub-metrics that combine specialist vision models with large multimodal models, and all metrics are validated against human judgments. Across 20 state-of-the-art models, we find that no single model performs strongly across all dimensions. We provide detailed diagnostic insights into the characteristic strengths, weaknesses, and open challenges of each model. Code and data are available at https://github.com/meituan-longcat/WBench.
Paper Collector 中文速览
首个交互式视频世界模型多轮评测基准
方法概述
构建包含多样场景的多轮交互数据集，设计22个自动子指标，结合专业视觉模型和大语言模型进行评估，所有指标均与人类判断验证。
核心贡献
提出WBench评测基准，包含289测试案例和1058交互轮次，从五个维度系统评估交互式世界模型，支持四种交互类型和多种控制方式。
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
How can interactive world models be systematically evaluated across multiple complementary dimensions to enable fair comparison and failure diagnosis?
核心方法
WBENCH evaluates 20 models across five dimensions using 22 fine-grained sub-metrics, all rescaled to [0, 100]. A dual-track protocol evaluates all models on a shared navigation subset (158 cases) while text-driven models are further evaluated on the full benchmark (289 cases). Navigation actions are mapped to text prompts, 6-DoF poses, or discrete commands depending on model paradigm to enable fair cross-paradigm comparison.
方法组件
- Twenty models are evaluated spanning three paradigms: text-driven (9), camera-controlled (5), and action-conditioned (6).
- Text-driven models are evaluated on the full 289-case test set via iterative last-frame forwarding.
- Camera-controlled and action-conditioned models are restricted to the navigation subset of 158 cases.
- Navigation actions are mapped to appropriate control signals (text, 6-DoF poses, or keyboard commands) for each paradigm to enable fair comparison.
- Action-conditioned models include both open and closed systems with varying architectures and parameter sizes.
- Matrix-Game 3.0 uses AR Diffusion 6B architecture with 1280×704 resolution and 70 frames.
- Models range from 1.5B to 13B parameters with different inference speeds.
- Both open models (Matrix-Game, HY-GameCraft, Infinite-World) and closed models (Happy Oyster, Genie 3) are evaluated.
- Text-driven models use an iterative I2V protocol where each turn extracts the last frame and constructs a combined text prompt.
- Cosmos-Predict 2.5 is NVIDIA's world foundation model pre-trained on diverse data sources.
论点验证
The paper provides concrete evidence of WBENCH's existence and implementation: 289 cases, 1,058 turns, detailed methodology across paragraphs 3-27, and systematic evaluation of 20 models with results in Table 2.
The paper explicitly enumerates five dimensions (Video Quality, Setting Adherence, Interaction Adherence, Consistency, Physical) and 22 sub-metrics (V.1-V.6, S.1-S.2, I.1-I.4, C.1-C.8, P.1-P.2) with detailed descriptions in paragraphs 21-27.
Table 2 presents results for 20 models across three paradigms. The automatic pipeline is described in detail (paragraphs 21-27) with specific metric implementations. Diagnostic insights are provided throughout paragraphs 29-39.
Paragraph 4 explicitly states the three aligned forms, and paragraphs 28, 51-54, and 74-84 detail how navigation is mapped to different interfaces for fair comparison.
Specific numbers are provided: 158 navigation cases for all 20 models, 289 cases and 1,058 turns for text-prompted I2V models. These are stated in paragraphs 4 and 17.
Paragraph 35 provides specific correlation values: navigation has near-zero correlation with video quality (r=-0.12), consistency (r=-0.05), and physical compliance (r=-0.15), demonstrating independence.
Paragraph 36 provides evidence: navigation has near-zero correlation with perspective consistency, and specific examples show top-navigation models (HY-World 1.5, Matrix-Game 3.0) rank lowest in perspective consistency.
Paragraph 35 provides specific correlations: physical scores correlate strongly with video quality (r=0.84) and consistency (r=0.72) but not navigation (r=-0.15).
Paragraph 38 provides specific z-scores for different categories: first-person z=+1.0, third-person harder; sports/game z=-1.9, animal subjects z=-1.9, workspace z=+1.6, robot subjects z=+1.0.
Paragraph 39 provides specific degradation values: navigation -33 points (turn 1 to turn 4+), event editing -13, subject action -9, perspective switching +2. Navigation clearly degrades fastest.
The claim 'near saturation' is not quantitatively defined. Table 2 shows video quality scores in the 77-82 range, but without a defined saturation threshold or comparison to theoretical maximum, 'near saturation' remains a qualitative assertion.
Table 2 and paragraph 30 provide exact scores: Seedance 1.5 at 82.1, Wan 2.7 at 81.5, LingBot-World at 78.9, Happy Oyster at 77.3.
Paragraph 31 provides exact scores for Setting Adherence: Wan 2.7 (91.4), Kling 3.0 (91.0), Happy Oyster (74.2), LingBot-World (72.6). The gap of ~17 points is substantial.
Paragraph 32 provides exact averages: camera-controlled (76.0), action-conditioned (77.7), text-driven (67.6). The ~10 point gap is accurate.
Paragraph 32 states YUME 1.5 achieves 72.0 navigation score among text-driven models. Table 2 would confirm this is the highest among text-driven models.
Paragraph 32 states perspective switching average is 30.7, and paragraph 39 confirms this is the lowest performing task.
Paragraph 33 states LingBot-World achieves highest overall consistency at 89.9.
... 共 70 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available - implementation details not accessible
- WBENCH benchmark dataset not publicly available (289 test cases)
- No random seeds specified for reproducible experiments
- Inference hyperparameters missing (temperature, guidance scale, number of diffusion steps)
- Training hyperparameters not provided (learning rate, batch size, epochs)
- Detailed evaluation metrics implementation not fully specified - only partial formulas shown
- Web interface protocol details referenced in Appendix B.2 but not included
- Text prompt templates for text-driven models not specified
- Action-to-paradigm mapping details incomplete (how navigation actions convert to 6-DoF poses, keyboard commands)
- Frame sampling and video segment assembly parameters not fully detailed
局限与证据边界
- The current test set focuses on discrete action sequences rather than continuous control
- The physical dimension relies partly on LMM-based evaluation whose reliability may degrade for subtle effects
- Expanding to additional domains and real-time evaluation are promising extensions
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-28T01:29:57+00:00 · 数据来源：Paper Collector