MEMLENS introduces the first multimodal long-term memory benchmark, evaluating 27 LVLMs and 7 memory agents across 789 questions at 32K-256K token contexts. Results show LVLMs degrade with context growth (top score 58.68% at 32K) while memory agents remain stable but lose visual fidelity.
论文原始摘要
Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.
Paper Collector 中文速览
首个系统评测LVLM多模态长期记忆能力的基准
方法概述
构建跨模态token计数方案的多会话对话基准，涵盖5种记忆能力、4种上下文长度；通过图像消融实验验证视觉必要性；系统评估长上下文LVLM和记忆增强代理的性能差异。
核心贡献
提出MEMLENS基准，包含789问题，评测27个LVLM和7个记忆代理，揭示长上下文模型和记忆代理的优劣与局限。
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
How can multimodal long-term memory capabilities be evaluated in large vision-language models and memory-augmented agents under a unified protocol that requires genuine cross-modal reasoning?
核心方法
MEMLENS constructs 789 questions across four standardized context lengths (32K-256K tokens) using a four-stage pipeline with entity abstraction to enforce cross-modal dependency. The benchmark evaluates five memory abilities (Information Extraction, Multi-Session Reasoning, Temporal Reasoning, Knowledge Update, Answer Refusal) on 27 LVLMs and seven memory-augmented agents using LLM-as-Judge accuracy with cross-validation.
方法组件
- LLMs were used only for stylistic refinement of paragraphs originally drafted by authors.
- Scientific content, methodology, and experimental design were fully authored without LLM assistance.
- LLM contributions were limited to polishing and do not constitute co-authorship.
- 27 LVLMs are evaluated, including closed-source (GPT-5.4, Claude, Gemini) and open-source families.
- Seven memory-augmented agents include three multimodal and four text-only pipelines.
- LVLMs are evaluated at 32K, 64K, and 128K; agents at all four lengths (32K-256K).
- Released checkpoints are used for RL-finetuned and LoRA-adapted models.
- Text-only agents use BLIP-2 captions to replace visual inputs.
- Context-robust models balance bidirectional churn (correct→wrong and wrong→correct) rather than exhibiting static resistance, explaining their flat accuracy trajectories.
- Cross-modality ablation shows MSR accuracy reaches 100% for GPT-5.4 and 90.21% for Gemini-3.1-Pro with oracle retrieval, confirming the MSR ceiling is retrieval-bounded.
论点验证
The cross-modal design is a core methodological contribution. The paper explicitly describes interleaved text-image sessions (p_11) and the cross-modal reasoning requirement enforced through entity abstraction (p_24, p_28). The image-ablation study (
The five memory abilities are explicitly defined with detailed descriptions in p_12-23. Each ability is broken into subtypes with specific reasoning operations targeted. The taxonomy is grounded in established memory research (Appendix I).
The four-stage pipeline is described in detail in p_25-26 with explicit component names and Figure 1 reference. The methodology section provides comprehensive documentation of each stage.
The four context lengths and cross-modal token-counting scheme are specified in p_7 and p_29, with reference to MMLongBench [11] for the counting methodology. Table 2 provides the implementation details.
The image-ablation study is reported with specific quantitative results: 80.4% of questions include images, and accuracy collapses below 2% when images are removed (p_7, p_32). Table 3 provides the detailed results. This is strong empirical evidence.
The four components are explicitly enumerated in p_25-26 with detailed descriptions of each stage. This is a methodological claim that is thoroughly documented.
Entity abstraction with 55 categories is specified in p_28 and p_67, with examples provided (e.g., 'Golden Gate Bridge' → 'the bridge shown in
The three-round human review process is documented in p_31 and p_99-116, with specific numbers: 20k candidates reduced to 789 questions. Inter-annotator agreement (κ = 0.78) is reported in p_105.
Specific quantitative results are provided: GPT-5.4 reaches 93.13% and Gemini-3.1-Pro reaches 89.42% on the answerability test with n=634 questions (p_32). This validates that questions are answerable when full evidence is provided.
The multimodal ablation result is reported with specific numbers: accuracy collapses below 2% for both models when evidence images are removed (p_32). This is the same empirical finding as claim 6.
Stated in p_34 as a finding from the evaluation. The claim about Gemini-3.1-Pro being competitive on IE, KU, and MSR at 128K is a specific comparative finding, though exact numerical thresholds for 'competitive' are not explicitly defined.
The finding is stated in p_34 with explanation of the mechanism (BM25 date matching for TR, keyword retrieval vs visual grounding for IE). This is a specific architectural analysis of Memory-T1's performance pattern.
Stated in p_35 as a key finding. The paper provides analysis explaining why IE and KU show the largest gaps (visually grounded types requiring fine-grained visual cues). The claim is supported by the evaluation results.
Specific quantitative results are provided: Mem0 (77.27%), MemOS (68.18%) for frozen-backbone agents, and 9-22% AR for finetuned agents (p_36). This is strong empirical evidence with exact numbers.
Specific quantitative comparison provided: M2A reaches 22.73% vs 81.82% under direct inference on the same Qwen3-VL-8B backbone (p_36). This is a controlled comparison with exact numbers.
... 共 58 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- Random seeds for evaluation reproducibility not specified
- Sampling parameters (temperature, top-p, etc.) for evaluated models and judge model not documented
- Hardware/compute specifications not reported
- API versions and access dates for closed-source models (GPT-5.4, Claude Sonnet 4.5, Gemini-3.1-Pro) not specified
- Number of evaluation runs and whether results are averaged across runs not stated
- Confidence intervals or error bars for reported accuracy metrics not provided
- Exact BLIP-2 captioning configuration and parameters not detailed
- Judge model (Qwen3-VL-235B-A22B-Instruct) inference parameters not specified
- Preprocessing pipeline details for image/text inputs not fully documented
局限与证据边界
- Many LVLMs do not natively support 256K context, so LVLMs are evaluated at three context lengths (32K, 64K, and 128K) for a fair comparison.
- MEMLENS is not intended as a training dataset: exposure of the 789 evaluation items as supervised data would compromise their diagnostic value.
- The reward design of current RL/SFT fine-tuning on memory agent backbones optimizes mainly answer correctness and retrieval success, providing no signal that refusing an unanswerable question is correct, so abstention degrades after training.
- Neither approach comes close to solving the task of long-term memory.
- The two approaches exhibit complementary failure modes: long-context LVLMs deliver high shortcontext accuracy through direct visual grounding, but this advantage shrinks as conversations grow; memory agents, in contrast, are length-stable but lose visual fidelity under storage-time compression.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-16T13:32:46+00:00 · 数据来源：Paper Collector