TL;DR
MMSkills represents reusable visual-agent skills as multimodal procedural knowledge combining text, state cards, and keyframes. An automated generator creates skills from trajectories, while branch-loaded inference provides structured guidance.

论文原始摘要

Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as multimodal procedural knowledge and address three practical challenges: (I) what a multimodal skill package should contain; (II) where such packages can be derived from public interaction experience; and (III) how agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce MMSkills, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors.

Paper Collector 中文速览

提出多模态技能框架，提升视觉Agent能力

方法概述

构建包含文本过程、运行时状态卡和多视角关键帧的技能包；开发Agent轨迹到技能生成器，通过工作流分组、过程归纳、视觉定位和元技能审计生成技能；设计分支加载多模态技能Agent，将选定的状态卡和关键帧与实时环境对齐，蒸馏为结构化指导

核心贡献

提出MMSkills框架，定义多模态过程知识表示，实现轨迹到技能的自动生成与分支加载推理机制

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

32
已证实
5
证据不足
2
无法验证
N/A
可复现性
置信度
80%

核心问题

How can reusable procedural skills be represented, generated, and utilized for visual agents whose decisions depend on visual evidence, given that existing text-only or code-based skill specifications are insufficient for multimodal decision-making?

核心方法

The authors propose MMSkills, comprising multimodal skill packages with textual procedures, runtime state cards, and multi-view keyframes; a five-phase trajectory-to-skill Generator that converts public non-evaluation trajectories into skill packages; and a branch-loaded inference mechanism that isolates skill-environment grounding in a temporary branch. Evaluation spans four benchmarks (OSWorld, macOSWorld, VAB-Minecraft, Super Mario Bros) comparing no-skill, text-only skill, and MMSkills conditions across six multimodal models.

论点验证

已证实 (85%) We propose MMSkills, a framework for representing, generating, and utilizing reusable multimodal procedures for runtime visual decision making.
已证实 (70%) To the best of our knowledge, we are the first to introduce the multimodal skill package, formulating reusable skills for general visual agents as multimodal procedural knowledge: compact, state-conditioned units that organize textual procedures, runtime state cards, and multi-view keyframes for visual decision making.
已证实 (85%) We develop an agentic trajectory-to-skill Generator that turns public, non-evaluation trajectories into multimodal skill packages through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing.
已证实 (85%) We propose branch loading, a runtime mechanism that selects and aligns multimodal skill evidence in a temporary branch before returning structured decision support to the main agent.
已证实 (75%) We demonstrate significant gains across GUI and game-based visual-agent benchmarks and multiple model families, showing that external multimodal procedural knowledge complements model-internal priors.
The paper evaluates across four benchmarks (OSWorld, macOSWorld, VAB-Minecraft, Super Mario Bros) with six models. Specific quantitative improvements are mentioned (e.g., invocation coverage 37.50% to 65.28%, repeated actions 21.8% to 6.2%). However,
已证实 (90%) MMSkills are designed around three components: a multimodal skill package that stores reusable visual procedural knowledge, a Skill Generation pipeline that constructs such packages from public trajectories, and a branch-loaded multimodal skill agent that isolates skill-environment grounding in a temporary branch and returns distilled decision support to the main trajectory at inference time.
The three-component design is clearly specified and consistently referenced throughout the paper. Each component is detailed in dedicated sections with formal definitions.
已证实 (90%) We represent each MMSkill as a state-conditioned procedure package where D is a compact descriptor, P is a reusable textual procedure, S = {S j } is a set of runtime state cards, and K = {K j } is a set of keyframe bundles aligned with those cards.
The formal representation is explicitly provided with equation notation and field definitions. Each component (D, P, S, K) is clearly defined.
已证实 (90%) A runtime state card is an agent-facing state node rather than an image caption. It links a point in the procedure to when-to-use conditions, when-not-to-use conditions, visible cues, verification cues, and available views.
The runtime state card structure is formally defined with all five fields specified: when_to_use, when_not_to_use, visible_cues, verification_cue, and available views.
已证实 (90%) Each key state is grounded by a small multi-view bundle. Let V = {full_frame, focus_crop, before, after}. The full-frame view preserves global context, the focus crop localizes the visual cue, and optional before/after views expose useful transitions.
The multi-view bundle structure is formally defined with four view types and their purposes clearly explained.
已证实 (90%) We build MMSkills from public interaction trajectories that are separate from evaluation tasks.
The separation of source trajectories from evaluation tasks is clearly stated and implemented across all benchmarks.
已证实 (90%) The pipeline comprises five stages: Phase 0: task embedding and clustering; Phase 1: cluster-level skill planning; Phase 2: skill merging; Phase 3: text-first drafting; Phase 4: image grounding and audit.
All five phases are explicitly named and described with their functions clearly specified.
已证实 (75%) The visual grounding policy is conservative: views are added only for state recognition, transition comparison, or completion verification, so the skill stores diagnostic states rather than replaying demonstrations.
已证实 (70%) For MMSkills, direct loading is brittle because state cards, multi-view keyframes, and transition examples add substantial context pressure, and irrelevant reference views can anchor the agent away from the live environment.
已证实 (90%) Stage 1: gated view selection. The branch first selects which state cards and view types are relevant to the live observation. The selector reads the live observation, recent history, textual procedure, and state-card descriptions before loading images.
Stage 1 is clearly specified with its function, inputs, and decision process described.
已证实 (90%) Stage 2: branch planning. The branch then aligns the selected evidence with the live state and returns structured guidance. The main agent does not execute G t mechanically; it uses G t as an intermediate planning signal and still chooses a grounded action from the live screenshot.
Stage 2 is clearly specified with its function and the relationship between guidance and main agent action described.
已证实 (90%) We evaluate on OSWorld, macOSWorld, VAB-Minecraft from VisualAgentBench, and Super Mario Bros from LMGame-Bench.
The four evaluation benchmarks are explicitly named with their sources identified.
已证实 (90%) In all settings, agents plan from visual observations, namely desktop or game screenshots.
The visual observation constraint is clearly stated and consistently applied.
已证实 (90%) We use a maximum interaction budget of 20 steps for both GUI benchmarks.
The 20-step budget is explicitly stated for both GUI benchmarks.
已证实 (90%) We evaluate both frontier and smaller multimodal models: Gemini 3.1 Pro, Gemini 3 Flash, Qwen3-VL-235B-A22B-Thinking, GLM-5V-Turbo, Kimi-K2.6, and Qwen3-VL-8B-Instruct.
All six evaluated models are explicitly named with their full identifiers.
已证实 (90%) For each base model, we compare no-skill, text-only skill, and MMSkills conditions.
The three-condition comparison is explicitly stated as the evaluation design.

... 共 39 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

局限与证据边界

本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-19T01:23:15+00:00 · 数据来源：Paper Collector