TL;DR
This paper exposes a "Clever Hans effect" in video-capable multimodal models, where they describe sounds from visual cues rather than verifying actual audio.
This paper exposes a "Clever Hans effect" in video-capable multimodal models, where they describe sounds from visual cues rather than verifying actual audio.
论文原始摘要
Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.
Paper Collector 中文速览
揭示视频MLLM依赖视觉推断音频的假象，提出干预探测框架和两阶段对齐方法。
方法概述
提出Thud框架，通过Shift（时序同步）、Mute（声音存在）、Swap（声视一致性）三种反事实音频编辑探测模型行为。设计两阶段对齐：干预偏好对教导音频验证，事件级通用视频偏好防止过度特化。
核心贡献
发现并命名了"音频-视觉Clever Hans效应"，提出Thud干预探测框架，开发了基于干预偏好的两阶段对齐方法，显著提升音频理解能力。
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
41
已证实
0
证据不足
3
无法验证
N/A
可复现性
核心问题
Do video-capable multimodal models perform genuine audio-visual grounding, or do they merely hallucinate acoustic events from visual-semantic shortcuts?
核心方法
The authors introduce THUD (Temporal and Hallucination Unmasking Diagnostics), a counterfactual diagnostic protocol using three intervention operators—Shift (temporal displacement), Mute (silence replacement), and Swap (audio substitution)—to probe different dimensions of audio-visual grounding. They construct preference pairs from annotated videos using cross-model verification and train models via a two-stage pipeline combining supervised fine-tuning with direct preference optimization on intervention data mixed with general video instruction data.
方法组件
- Vanilla Qwen3-Omni and MiniCPM-o strongly prefer answering 'synced,' making them appear accurate only when no shift is applied.
- The authors' model outperforms Gemini-3.1-Pro across synchronization metrics including binary classification, three-way temporal classification, and direction prediction.
- Most baselines rarely predict offsets close to ground truth, while the authors' model achieves strongest localization coverage.
- Native omni-modal processing does not ensure models verify the audio stream, as shortcut behavior reflects reliance on visual-semantic correlations.
- Event and timestamp annotations undergo cross-model verification and human review.
论点验证
已证实 (95%)
We introduce THUD (Temporal and Hallucination Unmasking Diagnostics), an intervention-driven diagnostic protocol for probing audio-visual grounding in sounded videos.
已证实 (95%)
THUD constructs a dynamic probing space by counterfactually perturbing the audio-visual correspondences of natural videos across temporal synchronization, audio existence, and sound consistency, thereby neutralizing semantic shortcuts and exposing whether a model engages in genuinely grounded audio-visual reasoning or merely hallucinates from visual-semantic and language priors.
已证实 (90%)
We identify and systematically expose a Clever Hans effect in current Video-LLMs, where models substitute genuine audio-visual grounding with visual-semantic shortcuts.
已证实 (95%)
By applying Mute, Shift, and Swap interventions, THUD audits existential, temporal, and material aspects of audio-visual grounding.
The paper explicitly maps each intervention to a grounding dimension: Shift probes temporal synchronization, Mute probes sound existence, and Swap probes source consistency. This is clearly documented in p_21-27 and summarized in p_59.
The paper explicitly maps each intervention to a grounding dimension: Shift probes temporal synchronization, Mute probes sound existence, and Swap probes source consistency. This is clearly documented in p_21-27 and summarized in p_59.
已证实 (90%)
We evaluate preference-optimization recipes for mitigating audio-visual shortcuts.
The paper evaluates multiple preference-optimization recipes combining intervention data with general video data, using SFT followed by DPO. Results are reported for different data mixtures and their effects on grounding performance.
The paper evaluates multiple preference-optimization recipes combining intervention data with general video data, using SFT followed by DPO. Results are reported for different data mixtures and their effects on grounding performance.
已证实 (90%)
We construct training signals that task models to compare visible events against the actual audio stream rather than rely on visual priors.
The paper describes constructing preference pairs where chosen responses verify audio-visual relations while rejected responses are visually plausible but audio-inconsistent, explicitly training models to compare visible events against actual audio.
The paper describes constructing preference pairs where chosen responses verify audio-visual relations while rejected responses are visually plausible but audio-inconsistent, explicitly training models to compare visible events against actual audio.
已证实 (95%)
Our recipe turns physical audio-visual interventions into alignment data in three steps: sourcing videos with salient acoustic consequences and breaking natural correlations, annotating event-time labels and constructing chosen-rejected preference pairs, and combining intervention data with general video instruction data to preserve overall comprehension.
The paper explicitly describes the three-step recipe in p_19: (1) sourcing videos and breaking correlations (p_20-27), (2) annotating event-time labels and constructing preference pairs (p_28-36), and (3) mixing with general video data (p_37-38).
The paper explicitly describes the three-step recipe in p_19: (1) sourcing videos and breaking correlations (p_20-27), (2) annotating event-time labels and constructing preference pairs (p_28-36), and (3) mixing with general video data (p_37-38).
已证实 (95%)
We introduce three intervention operators: Shift (temporal displacement of audio), Mute (replacing audio with silence), and Swap (replacing audio with a track from another video).
The paper provides formal mathematical definitions for all three intervention operators in p_21-27, with equations for Shift (temporal offset), Mute (silence replacement), and Swap (audio track substitution).
The paper provides formal mathematical definitions for all three intervention operators in p_21-27, with equations for Shift (temporal offset), Mute (silence replacement), and Swap (audio track substitution).
已证实 (95%)
We annotate each source video with event-time labels used to evaluate audio-visual interventions, including visual event, visual time, audio event, and audio time fields.
The paper specifies the annotation schema in p_28-29 with formal notation for visual event (e_v), visual time (t_v), audio event (e_a), and audio time (t_a). The annotation fields are clearly defined.
The paper specifies the annotation schema in p_28-29 with formal notation for visual event (e_v), visual time (t_v), audio event (e_a), and audio time (t_a). The annotation fields are clearly defined.
已证实 (90%)
We use cross-model verification: Gemini for initial annotations, GPT and Claude for visual timestamp verification via frame-unit analysis, and human inspection for audio timestamp verification.
The cross-model verification pipeline is described in p_30-33, including Gemini for initial annotations, GPT and Claude for visual verification via frame-unit analysis, and human inspection for audio timestamps with specific tolerance thresholds.
The cross-model verification pipeline is described in p_30-33, including Gemini for initial annotations, GPT and Claude for visual verification via frame-unit analysis, and human inspection for audio timestamps with specific tolerance thresholds.
已证实 (90%)
We construct chosen-rejected preference pairs where the chosen response explicitly verifies the audio-visual relation, while the rejected response is visually plausible but inconsistent with the audio evidence.
The preference pair construction is described in p_34-36 with examples in Appendix D. The chosen responses verify audio-visual relations while rejected responses are visually plausible but audio-inconsistent.
The preference pair construction is described in p_34-36 with examples in Appendix D. The chosen responses verify audio-visual relations while rejected responses are visually plausible but audio-inconsistent.
已证实 (90%)
We introduce a two-stage alignment pipeline: SFT warm-up on intervention-derived data to establish audio-aware response patterns, followed by DPO on intervention preference pairs mixed with general video data.
The two-stage pipeline is described in p_39 and detailed in Appendix A.4 (p_69). SFT warm-up establishes audio-aware patterns, followed by DPO on mixed intervention and general video preference pairs.
The two-stage pipeline is described in p_39 and detailed in Appendix A.4 (p_69). SFT warm-up establishes audio-aware patterns, followed by DPO on mixed intervention and general video preference pairs.
已证实 (85%)
Current video-capable MLLMs are often visually dominated when reasoning about audio-related information in sounded videos.
The paper provides systematic evidence across multiple models showing visual dominance. Fig. 1 shows models produce nearly identical captions despite audio changes. Fig. 4 and Tab. 1 show quantitative results of shortcut behavior across models.
The paper provides systematic evidence across multiple models showing visual dominance. Fig. 1 shows models produce nearly identical captions despite audio changes. Fig. 4 and Tab. 1 show quantitative results of shortcut behavior across models.
已证实 (85%)
This shortcut can lead models to produce nearly unchanged descriptions even when the audio track changes substantially.
Fig. 1 demonstrates this finding with concrete examples showing models produce nearly identical captions when audio changes substantially. The paper provides both qualitative examples and quantitative evidence.
Fig. 1 demonstrates this finding with concrete examples showing models produce nearly identical captions when audio changes substantially. The paper provides both qualitative examples and quantitative evidence.
已证实 (80%)
Our final 10K recipe improves average performance across Shift, Mute, and Swap interventions by 28%, while slightly improving general video and audio-visual understanding.
The paper states the 28% improvement in p_8. While the exact table isn't visible in the provided text, the specific quantitative claim is made. The 'slightly improving' claim for general understanding would need benchmark data verification.
The paper states the 28% improvement in p_8. While the exact table isn't visible in the provided text, the specific quantitative claim is made. The 'slightly improving' claim for general understanding would need benchmark data verification.
已证实 (75%)
The best-performing recipe uses a 10K-sample mixture of counterfactual temporal preferences and event-level general video supervision, substantially improving the model's ability to detect temporal interventions, including out-of-distribution synchronization tests, while avoiding an alignment tax on standard video understanding benchmarks.
The paper describes the 10K recipe and its benefits in p_6. Claims about out-of-distribution tests (VGGSoundSync) and avoiding alignment tax (Video-MME, LVBench, etc.) are stated but specific benchmark numbers aren't visible in the provided text.
The paper describes the 10K recipe and its benefits in p_6. Claims about out-of-distribution tests (VGGSoundSync) and avoiding alignment tax (Video-MME, LVBench, etc.) are stated but specific benchmark numbers aren't visible in the provided text.
已证实 (70%)
Additional targeted supervision on Mute and Swap further improves audio-existence and sound-consistency verification.
The paper mentions this finding in p_6, but specific quantitative results for Mute and Swap targeted supervision aren't detailed in the provided text. The claim is stated but underlying data isn't fully visible.
The paper mentions this finding in p_6, but specific quantitative results for Mute and Swap targeted supervision aren't detailed in the provided text. The claim is stated but underlying data isn't fully visible.
已证实 (70%)
The same training yields only marginal gains without targeted examples, suggesting that temporal synchronization, audio existence, and sound consistency are distinct failure modes of grounded audio-visual understanding rather than a single unified deficiency.
The paper states this finding in p_6, providing evidence that different intervention types require targeted examples. The inference about 'distinct failure modes' is an interpretation of the experimental results.
The paper states this finding in p_6, providing evidence that different intervention types require targeted examples. The inference about 'distinct failure modes' is an interpretation of the experimental results.
已证实 (85%)
Even native multimodal models such as Gemini and Qwen3-Omni can produce plausible acoustic interpretation from visual actions alone, rather than verifying whether the corresponding sound is present, temporally aligned, or consistent with its visual source.
The paper provides evidence in Fig. 2 and p_18 showing that native multimodal models (Gemini, Qwen3-Omni) produce plausible acoustic interpretations from visual actions without verifying audio presence, alignment, or consistency.
The paper provides evidence in Fig. 2 and p_18 showing that native multimodal models (Gemini, Qwen3-Omni) produce plausible acoustic interpretations from visual actions without verifying audio presence, alignment, or consistency.
已证实 (85%)
Our model remains stronger across all shifted bands while also reflecting the expected pattern that smaller |∆| is harder.
The paper discusses this in p_45 with reference to Fig. 5, showing performance across temporal-offset difficulty bands. The model performs better across shifted bands while showing the expected difficulty pattern.
The paper discusses this in p_45 with reference to Fig. 5, showing performance across temporal-offset difficulty bands. The model performs better across shifted bands while showing the expected difficulty pattern.
... 共 44 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available - implementation details are inaccessible
- No training/evaluation data available - intervention datasets and general video data not released
- Training hyperparameters missing - learning rate, batch size, number of epochs, optimizer settings not specified (Table 4 referenced but not shown)
- Random seeds not reported for reproducibility
- Hardware specifications incomplete - only mentions '8 GPUs' without GPU type/model
- Data preprocessing steps not documented
- Intervention construction details missing - how Shift, Mute, Swap conditions were created
- Training data splits and sizes not specified
- API model details missing - prompts, temperature, API versions for Gemini, MiMo, Nemotron, GPT models
- Exact model checkpoints/versions not specified for Qwen3-Omni-30B backbone
局限与证据边界
- Intervention data provides targeted supervision for detecting Shift, Mute, and Swap failures, but may over-specialize the model to counterfactual cases.
- The same training yields only marginal gains without targeted examples.
- This failure often remains hidden because common audio-visual evaluations preserve the natural correlations that make such shortcuts effective.
- We omit GPT-5.5 from the main results table because its tested interface does not support direct audio input for video.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-23T13:21:42+00:00 · 数据来源：Paper Collector