TL;DR
The paper introduces MMProLong, a method for training long-context vision-language models using carefully designed long-document VQA data. Using a 1.5M PDF document pool and a short-to-long synthesis pipeline, the authors train Qwen2.

论文原始摘要

Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in agentic workflows. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; and iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.1% and maintains strong performance at 256K and 512K contexts beyond its 128K training window, without additional training. It further generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision. Overall, our study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models.

Paper Collector 中文速览

系统研究LVLM长上下文训练，提出MMProLong模型

方法概述

基于Qwen2.5-VL-7B进行5B token长上下文持续预训练，通过消融实验发现：长文档VQA优于OCR转录、平衡长度分布优于目标长度聚焦、检索是主要瓶颈、纯长文档VQA可保留短上下文能力

核心贡献

系统研究长上下文训练配方，提出MMProLong模型，在128K训练窗口内实现512K外推能力，长文档VQA提升7.1%

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

43
已证实
4
证据不足
2
无法验证
N/A
可复现性
置信度
83%

核心问题

How can large vision-language models be effectively trained to handle long-context inputs of 128K tokens and beyond while maintaining strong generalization capabilities?

核心方法

The authors extend Qwen2.5-VL-7B's context window from 32K to 128K tokens using Dynamic-NTK scaling and train with a 5B token budget. They construct a document pool of 1.5M PDFs and develop a short-to-long synthesis pipeline that generates long-document VQA training data from semantically coherent page segments. The approach is evaluated through systematic ablations comparing different training tasks, data mixtures, and length distributions.

论点验证

已证实 (95%) We conduct our LongPT experiments using Qwen2.5-VL-7B, extending its original 32K context window to 128K.
已证实 (90%) Following the Dynamic-NTK heuristic, we scale the mRoPE base frequency from its original value of 1 × 10^6 to 4 × 10^6.
The design choice is clearly stated and justified by an ablation study (Table 16) that tests mRoPE bases of 2×10^6, 4×10^6, and 8×10^6, showing that 4×10^6 achieves good performance.
已证实 (95%) Each LongPT run is trained with a fixed budget of 5B tokens, a maximum sequence length of 131,072 tokens, and a global batch size of 4M tokens.
Clear specification of training hyperparameters with exact numerical values. These are methodological parameters that define the experimental setup.
已证实 (85%) We first construct a large-scale document pool comprising over 1.5 million PDF-formatted documents from multiple sources.
The paper states the document pool size clearly (over 1.5 million PDFs). Appendix C.1 is referenced for detailed statistics. This is a factual claim about data collection.
已证实 (90%) For data synthesis, we select documents with 32 to 50 pages from this pool. With the 2×2-pixel unshuffle in our Qwen2.5-VL backbone, these documents yield multimodal sequences ranging from 32K to 128K tokens.
Clear design choice with specific parameters (32-50 pages) and reasoning about resulting token range (32K-128K). The connection to the 2×2-pixel unshuffle is explained.
已证实 (85%) To avoid evaluation contamination, we further filter out potential overlap with evaluation benchmarks using SHA-256 hashes of PDF content.
Clear design choice for data quality control using SHA-256 hashes. This is a standard deduplication/contamination prevention method.
证据不足 (50%) Each PDF page is rendered to an image at DPI = 144 using PyMuPDF. This resolution provides a practical trade-off between visual fidelity and storage cost.
已证实 (85%) We use an OCR expert model fine-tuned from Seed 2.0 to parse each rendered page into layout-aware blocks.
Clear specification of the OCR model used (fine-tuned from Seed 2.0) and its purpose (parsing into layout-aware blocks).
已证实 (90%) We construct the long-document VQA training data using a short-to-long synthesis pipeline. The key idea is to generate a QA pair from a short, semantically coherent page segment, and then place it back into the full-document context to form a long-context training instance.
The short-to-long synthesis pipeline is clearly described with the key idea explained. The pipeline is detailed in p_7-8 with examples.
已证实 (90%) We randomly sample one or more consecutive sections whose total length spans 8-15 pages. This produces a coherent page segment at the section level for QA generation.
Clear design choice with specific parameters (8-15 pages for segment length). The reasoning (section-level coherence) is explained.
已证实 (85%) We find that the generated QA pairs are of high quality, and further verify them through a manual check described in Appendix D.4.
Quantitative evidence provided: 97 out of 100 randomly sampled QA pairs are fully correct (p_68). This is concrete evidence of data quality.
已证实 (85%) To avoid such global-context false positives, we require the QA-generation model to add explicit segment anchors to the question, such as "in the Introduction section" or "on pages 20-25".
Clear design choice with motivation (avoiding global-context false positives) and examples of segment anchors.
已证实 (90%) We synthesize three training tasks of long-document VQA data, each targeting a distinct capability defined by the type and number of evidence pieces required to answer the question: (i) single-page extraction (extract-single), (ii) multi-page extraction (extract-multi), and (iii) reasoning.
The three training tasks are clearly defined with examples for each: extract-single, extract-multi, and reasoning. Each targets distinct capabilities.
已证实 (85%) Another category of long-context training tasks we build is OCR transcription. This task category encourages LVLMs to capture long-distance image-text dependencies by requiring them to transcribe text elements across all pages of a long document.
OCR transcription task category is clearly described with its purpose (capturing long-distance image-text dependencies).
已证实 (90%) We compare the five candidate tasks under a controlled 5B-token budget. For each task, we build a separate training set and train Qwen2.5-VL-7B using the hyperparameters in Section 3.
Clear experimental setup with controlled conditions (5B-token budget, separate training sets, fixed hyperparameters).
已证实 (95%) The 32K base model degrades substantially at 128K, with MMLongBench-Doc dropping from 32.17% to 26.96%.
Specific quantitative evidence: MMLongBench-Doc drops from 32.17% to 26.96% when evaluating 32K base model at 128K context.
已证实 (95%) OCR transcription tasks yield poor downstream performance, especially full-document OCR, whose overall average drops by 17.4% to 33.17%.
Specific quantitative evidence: full-document OCR overall average drops by 17.4% to 33.17%.
已证实 (95%) After adding the SFT stage to improve instruction-following ability, the OCR-trained checkpoints obtain moderate gains of 3.24% and 1.85% for full-document and needle-page OCR, respectively.
Specific quantitative evidence: SFT stage yields gains of 3.24% and 1.85% for full-document and needle-page OCR respectively.
已证实 (95%) All three long-document VQA tasks consistently improve performance by more than 5% in absolute terms, with multi-page extraction achieving the best average of 56.90%.
Specific quantitative evidence: all three long-document VQA tasks improve by >5% absolute, with multi-page extraction achieving 56.90% average.
已证实 (85%) Long-document VQA is a stronger and more computationally efficient supervision source for LongPT, yielding better downstream performance without an additional 5B-token SFT stage.
Conclusion well-supported by Table 1 results showing long-document VQA outperforms OCR transcription (which required additional SFT stage). The efficiency claim is justified by not needing the extra 5B-token SFT.

... 共 49 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

局限与证据边界

本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-15T01:09:59+00:00 · 数据来源：Paper Collector