Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? - AI 论文深度分析

TL;DR
The paper introduces Grounded Personality Reasoning and MM-OCEAN benchmark to evaluate MLLMs' personality perception. Testing 27 models reveals a "Prejudice Gap": 51.3% of correct ratings lack grounded evidence, with only 10.

论文原始摘要

Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with three contributions. (i) A new task: we formalize Grounded Personality Reasoning (GPR), which requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding. (ii) A new dataset: we release MM-OCEAN (1,104 videos, 5,320 MCQs), produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs. (iii) Benchmark and analysis: we design a three-tier evaluation (rating, reasoning, grounding) plus four sample-level failure-mode metrics: Prejudice Rate (PR), Confabulation Rate (CR), Integration-failure Rate (IR), and Holistic-grounding Rate (HR), and benchmark 27 MLLMs (13 closed, 14 open). The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%. These findings expose a disconnect between getting the right score and reasoning for the right reason, charting a roadmap for grounded social cognition in MLLMs.

Paper Collector 中文速览

提出GPR任务和MM-OCEAN数据集，揭示MLLM人格判断存在偏见

方法概述

提出基于证据的人格推理任务（GPR），通过评分-推理-锚定三阶段流程；构建多代理流水线生成MM-OCEAN数据集，包含1104个视频和5320道多选题；设计三层评估加四种失败模式指标（偏见率、幻觉率等）来全面评估27个MLLM

核心贡献

提出GPR任务框架，构建MM-OCEAN视频数据集，设计三层评估体系，揭示51%正确判断缺乏证据支撑的偏见现象

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

已证实

证据不足

无法验证

N/A

可复现性

置信度

90%

核心问题

Can Multimodal Large Language Models genuinely perceive personality traits from observable behavioral evidence, or do they merely rely on superficial prejudices and correlations to produce correct ratings without proper grounding?

核心方法

The authors construct MM-OCEAN benchmark with 1,104 videos and 5,320 cue-grounding MCQs through a five-stage multi-agent human-collaborative annotation pipeline using four LLM agents (Observer, Psychologist, Examiner, Aligner) with human verification. The evaluation framework comprises three tasks of increasing cognitive depth—ordinal personality rating (T1), open-ended reasoning (T2), and structured cue grounding (T3)—with cross-task diagnostic rates to identify specific failure modes in the reasoning chain.

方法组件

The evaluation framework consists of three tasks of increasing cognitive depth.
Task 1 focuses on ordinal personality rating, Task 2 on open-ended reasoning, and Task 3 on structured cue grounding.
Cross-task diagnostic rates identify specific failure points in the reasoning chain.
The framework goes beyond aggregate scores to provide sample-level failure-mode analysis.
27 MLLMs from 12 families are evaluated, split between 13 proprietary and 14 open-source models.
Uniform sampling and identical structured prompts ensure fair comparison across all models.
GPT-4o-mini serves as the AI-as-Judge for Task 2, with robustness verified across Claude Haiku 4.5 and Gemini 2.5 Flash-Lite.
Cross-judge ranking stability is confirmed with Spearman ρ ≥ 0.92.
Every evaluated MLLM shows positive deviation on reasoning cluster and negative deviation on visual-grounding cluster.
This confirms a field-wide preference for semantic reasoning over fine-grained perceptual localization.

论点验证

已证实 (95%) We introduce Grounded Personality Reasoning (GPR), which requires a model to (1) perceive fine-grained multimodal behavioral cues, (2) reason about how these cues map to personality traits via evidence-based analysis, and (3) demonstrate these abilities on structured multiple-choice probes that target specific sub-skills.

已证实 (95%) We construct MM-OCEAN, comprising 1,104 videos and 5,320 cue-grounding MCQs built by a five-stage multi-agent human-collaborative annotation pipeline.
The dataset statistics are explicitly reported in paragraph 21: 1,104 test videos and 5,320 cue-grounding MCQs. The five-stage pipeline is detailed in paragraphs 14-19 with clear description of each stage. The numbers match the claim exactly.

已证实 (92%) We add four sample-level failure-mode rates: Prejudice rate (PR; right rating, wrong cues), Confabulation rate (CR; plausible rationale, wrong cues), Integration-failure rate (IR; right cues, wrong rating), and Holistic-Grounding rate (HR; all three correct).

已证实 (92%) 51% of all correct ratings come without grounded cue retrieval, and the Holistic-Grounding Rate spans only 0-33.5%.

已证实 (88%) Even at the closed-source frontier, ~15% of correct ratings remain ungrounded.
Paragraph 39 reports: 'Even at the proprietary frontier (Gemini 3 Flash, GPT-5.5, Gemini 3.1 Pro), Top-3 mean PR ≈ 14.5%'. The ~15% figure is a reasonable approximation of 14.5%. The specific models and exact percentage are provided.

已证实 (90%) We formalize Grounded Personality Reasoning (GPR), distinguishing genuine perception from prejudice via a rating-reasoning-grounding chain.

已证实 (92%) We release MM-OCEAN (1,104 videos, 5,320 MCQs) with timestamped atomic observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs, produced by an Observer-Psychologist-Examiner-Aligner pipeline with human verification.
All stated components are verified in the paper: 1,104 videos and 5,320 MCQs (p_21), timestamped observations (p_15), seven MCQ categories (p_17, Table 2), and the four-agent pipeline with human verification (p_14-p_19).

已证实 (95%) The grounding constraint E_i ⊆ {1, . . . , K} - every trait judgment must cite at least one observed cue - is what distinguishes GPR from Apparent Personality Recognition (APR), which evaluates only ŷi.

已证实 (92%) MM-OCEAN is constructed through a five-stage pipeline that interleaves four LLM agents (Observer, Psychologist, Examiner, and Aligner) with two complementary human roles: 24 trained annotator-verifiers (Stage 1) and a pool of expert reviewers (Stage 5).
The five-stage pipeline is detailed in paragraphs 14-19. Four LLM agents are named (Observer, Psychologist, Examiner, Aligner). Human roles are specified: 24 annotators in Stage 1 (p_15, p_47) and expert reviewers in Stage 5 (p_19, p_47).

已证实 (90%) 78.2% of Observer drafts are accepted, 14.6% corrected, and 5.9% deleted; pairwise verdict agreement on overlap pool is 77%.
Paragraph 15 reports: '78.2% of Observer drafts are accepted, 14.6% corrected, and 5.9% deleted; pairwise verdict agreement on overlap pool is 77%.' These are specific quantitative results from the annotation process.

已证实 (90%) The Examiner consumes the verified observations and Psychologist analyses and generates seven cue-grounding MCQs spanning a cognitive taxonomy organized from reasoning to visual grounding.

已证实 (90%) Each MCQ has six options: one correct answer and five distractors covering three failure modes (text-derivable, plausible-but-wrong-segment, near-miss).
Paragraph 17 explicitly states: 'Each MCQ has six options: one correct answer and five distractors covering three failure modes (text-derivable, plausible-but-wrong-segment, near-miss).' This is a clear design specification.

已证实 (88%) Cross-judge robustness validation via Claude 4.5/Gemini 2.5 confirms stable T2 ranking (ρ ≥ 0.92).
Paragraph 18 states: 'Cross-judge robustness validation via Claude 4.5/Gemini 2.5 confirms stable T2 ranking (ρ ≥ 0.92, App. J).' The specific correlation threshold and judges are named. Appendix J provides additional details.

已证实 (90%) Every MCQ is answered by two text-only LLMs (GPT-4o-mini and Gemini Flash) using only the question stem and options (no video, no observations); items that both LLMs answer correctly are flagged as transcript-derivable and dropped, ensuring every retained question requires multimodal grounding.

已证实 (90%) MM-OCEAN draws its videos from the ChaLearn First Impressions V2 dataset, which contains ~10K fifteen-second clips of single-person speech with crowd-sourced Big Five trait scores and ASR-extracted transcriptions.
Paragraph 20 states: 'MM-OCEAN draws its videos from the ChaLearn First Impressions V2 dataset [14], which contains ~10K fifteen-second clips of single-person speech with crowd-sourced Big Five trait scores and ASR-extracted transcriptions.'

已证实 (92%) The released benchmark comprises 1,104 test videos accompanied by three layers of fine-grained annotations: ~13.5K human-verified atomic behavioral observations across four perceptual channels (Expression, Action, Audio, Background); 5,520 trait-level personality analyses; and 5,320 cue-grounding MCQs (averaging 4.8 retained per video after filter).

已证实 (92%) MM-OCEAN evaluates each model through three tasks of increasing cognitive depth: ordinal personality rating (T1), open-ended rating reasoning (T2), and structured cue grounding (T3).

已证实 (90%) An AI-as-Judge J evaluates models output against GT along four dimensions: Evidence Coverage, Logical Coherence, Grounding Accuracy, and Directional Accuracy.
Paragraph 26 states: 'An AI-as-Judge J evaluates models output against GT along four dimensions: Evidence Coverage, Logical Coherence, Grounding Accuracy, and Directional Accuracy.' The scoring protocol is described in paragraphs 26-27.

已证实 (88%) We refer to the field-wide phenomenon that most 'correct' ratings come without grounded evidence - captured jointly by high PR, low HR, and within-model rating-vs-grounding rank disconnect RGM > 0 - as the Prejudice Gap.

已证实 (85%) A 3×3×3 threshold sweep confirms that the HR ranking is stable (ρ ≥ 0.92 across all 27 combos).
Paragraph 34 states: 'A 3×3×3 threshold sweep confirms that the HR ranking is stable (ρ ≥ 0.92 across all 27 combos; Appendix I).' The specific experimental design and result are reported, though full details are in appendix.

... 共 50 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

No code repository available - critical implementation details inaccessible
No MM-OCEAN dataset available - cannot access evaluation benchmark
Structured prompts for T1, T2, T3 not provided despite being central to methodology
Frame sampling strategy details missing - number of frames per video, sampling method
API parameters for proprietary models (temperature, top-p, max_tokens) not specified
vLLM serving configuration for open-source models not documented
Random seeds not reported for reproducibility
Dataset statistics missing - total samples, train/test splits, category distributions
AI-as-Judge prompt templates and evaluation criteria not provided
Hardware specifications referenced in Appendix Z but not accessible

局限与证据边界

MM-OCEAN focuses on apparent personality from short, single-speaker English video clips; throughout this work, this denotes the specific construct from First Impressions V2.
We evaluate Task 2 reasoning quality via an AI-as-Judge protocol.
Natural extensions include cross-cultural and multilingual videos, multi-judge ensembles for Task 2 reliability, and richer grounding operationalizations beyond MCQ-based cue retrieval.
MM-OCEAN inherits the cultural and linguistic biases of ChaLearn First Impressions V2, which is predominantly composed of Western-context English speakers.
Task 3 operationalizes grounding as MCQ-based retrieval over a predefined cue set; a high Prejudice Rate may in part reflect MCQ-design choices rather than a model's general inability to ground its judgment in observable behavior.

本分析由 PDF 阅读助手自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-23T01:28:13+00:00 · 数据来源：Paper Collector