TL;DR
SkillOpt introduces text-space optimization of agent skills for adapting frozen frontier models without weight updates. Using iterative refinement with bounded edits and validation gates, it achieves best or tied-best on all 52 evaluated cells across six benchmarks and seven models, with +23.

论文原始摘要

Agent skills today are hand-crafted, generated one-shot, or evolved through loosely controlled self-revision, none of which behaves like a deep-learning optimizer for the skill, and none of which reliably improves over its starting point under feedback. We argue the skill should instead be trained as the external state of a frozen agent, with the same discipline that makes weight-space optimization reproducible. SkillOpt is, to our knowledge, the first systematic controllable text-space optimizer for agent skills: a separate optimizer model turns scored rollouts into bounded add/delete/replace edits on a single skill document, and an edit is accepted only when it strictly improves a held-out validation score. A textual learning-rate budget, rejected-edit buffer, and epoch-wise slow/meta update make skill training stable while adding zero inference-time model calls at deployment. Across six benchmarks, seven target models, and three execution harnesses (direct chat, Codex, Claude Code), SkillOpt is best or tied on all 52 evaluated (model, benchmark, harness) cells and beats every per-cell competitor among human, one-shot LLM, Trace2Skill, TextGrad, GEPA, and EvoSkill skills. On GPT-5.5 it lifts the average no-skill accuracy by +23.5 points in direct chat, by +24.8 inside the Codex agentic loop, and by +19.1 inside Claude Code. Transfer experiments further show that optimized skill artifacts retain value when moved across model scales, between Codex and Claude Code execution environments, and to a nearby math benchmark without further optimization. Code: https://aka.ms/skillopt

Paper Collector 中文速览

首个可控的Agent技能文本空间优化方法

方法概述

将技能作为冻结Agent的外部状态，用独立优化器模型根据评分rollouts生成增删改编辑，仅接受严格提升验证分数的编辑。结合文本学习率预算、拒绝编辑缓冲区和epoch级慢更新机制，实现稳定训练且推理零开销。

核心贡献

提出SkillOpt，首个系统可控的文本空间优化器，通过验证驱动的编辑策略训练技能文档，在52个评测单元中全部达到最优或并列最优。

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

54
已证实
0
证据不足
4
无法验证
N/A
可复现性
置信度
89%

核心问题

Can frozen frontier language models be effectively adapted to new domains through text-space optimization of agent skills, treating skill documents as trainable state without modifying model weights?

核心方法

SkillOpt uses a frontier optimizer model to propose structured add/delete/replace edits to skill documents based on rollout evidence from the target model. The system applies bounded updates under a textual learning-rate budget, uses minibatch reflection to analyze failures and successes separately, validates candidates on a held-out selection split before acceptance, and maintains rejected-edit buffers and epoch-wise slow/meta updates for longer-horizon learning.

方法组件

论点验证

已证实 (95%) We introduce SkillOpt, a text-space optimizer for agent skills. Given a target domain, an initial skill, and the model being adapted, SkillOpt repeatedly samples trajectory batches, analyzes successes and failures, and asks a frontier optimizer model to propose structured add/delete/replace edits.
已证实 (95%) We formulate agent-skill learning as optimization over an external natural-language state and introduce SkillOpt, a harness-agnostic optimizer with rollout batches, reflection minibatches, add/delete/replace edits, textual learning rates, schedules, held-out acceptance, rejected-edit buffers, and epoch-wise slow/meta update.
The paper fully specifies all listed components: rollout batches (p_16), reflection minibatches (p_19), add/delete/replace edits (p_19, p_22), textual learning rates (p_21), schedules (p_21), held-out acceptance (p_24), rejected-edit buffers (p_25),
已证实 (90%) We provide a broad empirical study across six benchmarks, seven target models, and three execution harnesses, showing that SkillOpt is best or tied-best on 52 of 52 cells and outperforms no-skill, human-skill, one-shot LLM-skill, prompt-optimization (TextGrad, GEPA), and skill-evolution (Trace2Skill, EvoSkill) baselines under every model.
The paper provides extensive empirical evidence across 6 benchmarks (SearchQA, SpreadsheetBench, OfficeQA, DocVQA, LiveMathematicianBench, ALFWorld), 7 target models (GPT-5.5, GPT-5.4, GPT-5.4-mini, GPT-5.4-nano, GPT-5.2, Qwen3.5-4B, Qwen3.6-35B-A3B)
已证实 (90%) We validate the optimization design through component ablations and three forms of transfer (cross-model, cross-harness, cross-benchmark), showing that the exported skill artifact is compact, reusable, and deployable without model-weight updates.
已证实 (85%) Out of 52 evaluated (model, benchmark, harness) cells, SkillOpt is the best or tied-best measured method on all 52.
已证实 (95%) With GPT-5.5 in direct chat, it lifts SearchQA from 77.7 to 87.3, SpreadsheetBench from 41.8 to 80.7, OfficeQA from 33.1 to 72.1, DocVQA from 78.8 to 91.2, LiveMathematicianBench from 37.6 to 66.9, and ALFWorld from 83.6 to 95.5 (a +23.5 point average gain over no skill).
已证实 (90%) It also beats the strongest per-cell baseline drawn from human-written, one-shot LLM, Trace2Skill, TextGrad, GEPA, and EvoSkill skills by +5.4 points on average.
已证实 (95%) The same optimization interface is effective inside Codex-style and Claude Code-style execution loops, lifting GPT-5.5 by +24.8 and +19.1 points over no skill respectively, and outperforming EvoSkill by +14.0 and +3.2 points.
Exact numerical values are provided in p_5 and p_34: Codex harness +24.8 over no skill and +14.0 over EvoSkill; Claude Code harness +19.1 over no skill and +3.2 over EvoSkill. These are specific, verifiable numbers.
已证实 (90%) A SpreadsheetBench skill trained on GPT-5.4 improves every smaller GPT variant we test; a Codex-trained spreadsheet skill transfers to Claude Code with a +59.7 point gain; and an OlympiadBench skill yields positive gains on Omni-MATH.
已证实 (85%) Bounded textual learning outperforms uncontrolled rewriting, held-out gating prevents harmful proposals from accumulating, the rejected-step buffer converts failed edits into negative feedback, and the epoch-wise slow/meta update improves long-horizon refinement without bloating the deployed skill.
已证实 (90%) The learned skills remain compact (300-2,000 tokens after only 1-4 accepted edits), inspectable, and procedural rather than instance-specific.
已证实 (95%) On GPT-5.5 direct chat, the six-benchmark average rises from 58.8 (no skill) to 82.3 (SkillOpt), a +23.5 point absolute improvement, while the best per-cell baseline averages only 76.9, leaving SkillOpt +5.4 points clear of an oracle baseline that picks the best of six competing methods per cell.
已证实 (95%) Procedural benchmarks see the largest improvements: SpreadsheetBench 41.8→80.7, OfficeQA 33.1→72.1, and LiveMathematicianBench 37.6→66.9 on GPT-5.5; SpreadsheetBench 9.3→23.9 (×2.6) on Qwen3.5-4B; and ALFWorld 34.3→69.4 (×2.0) on GPT-5.4-nano.
All numerical values are provided in p_33 and match the claim exactly: SpreadsheetBench 41.8→80.7, OfficeQA 33.1→72.1, LiveMathematicianBench 37.6→66.9 on GPT-5.5; SpreadsheetBench 9.3→23.9 on Qwen3.5-4B; ALFWorld 34.3→69.4 on GPT-5.4-nano.
已证实 (95%) On the Codex harness, SkillOpt is best on all five evaluated benchmarks for GPT-5.5, with average gain +24.8 points over no skill and +14.0 over the next-best baseline (EvoSkill).
Exact numerical values provided in p_34: 'On the Codex harness, SkillOpt is best on all five evaluated benchmarks for GPT-5.5, with average gain +24.8 points over no skill and +14.0 over the next-best baseline (EvoSkill).'
已证实 (95%) On the Claude Code harness, it is best on all five benchmarks for GPT-5.5, with average gain +19.1 over no skill and +3.2 over EvoSkill.
Exact numerical values provided in p_34: 'On the Claude Code harness, it is best on all five benchmarks for GPT-5.5, with average gain +19.1 over no skill and +3.2 over EvoSkill.'
已证实 (90%) SpreadsheetBench climbs 47.5→78.0 and LiveMathematicianBench climbs 59.1→70.5 as the optimizer sees 1→100% of the training partition, while SearchQA saturates at roughly 84-86 after 20% already.
Specific numerical values provided in p_39: SpreadsheetBench climbs 47.5→78.0, LiveMathematicianBench climbs 59.1→70.5 as optimizer sees 1→100% of training partition. SearchQA saturates at 84-86 after 20%.
已证实 (90%) Varying the reflection mini-batchsize from 1 to 32 keeps SearchQA inside 85.9-87.1 and SpreadsheetBench inside 75.4-77.9, with the default B m =8 at or near the top on all three benchmarks.
Specific numerical ranges provided in p_39: varying reflection mini-batch size from 1 to 32 keeps SearchQA inside 85.9-87.1 and SpreadsheetBench inside 75.4-77.9, with default B_m=8 at or near the top.
已证实 (90%) Moving from B=8 to a full epoch keeps SearchQA inside 85.1-87.2 and SpreadsheetBench inside 75.0-77.5.
Specific numerical ranges provided in p_39: moving from B=8 to full epoch keeps SearchQA inside 85.1-87.2 and SpreadsheetBench inside 75.0-77.5.
已证实 (90%) Sweeping L t ∈ {1, 2, 4, 8, 16} shows that small or moderate edit budgets are competitive throughout: L t =4 achieves 86.5/78.2/56.5, the highest LiveMath score belongs to L t =8 at 66.9, and the lowest score across all five settings is still only 85.5 on SearchQA.
Specific numerical values provided in p_40: L_t=4 achieves 86.5/78.2/56.5, highest LiveMath score is L_t=8 at 66.9, lowest score across settings is 85.5 on SearchQA.
已证实 (95%) Constant decay scores 87.3/80.7/62.1, cosine 87.1/77.5/61.3, and linear 87.2/72.9/62.9, so the bounded-update story does not depend on a single specific scheduler.
Exact numerical values provided in p_40: constant decay 87.3/80.7/62.1, cosine 87.1/77.5/61.3, linear 87.2/72.9/62.9.

... 共 58 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

局限与证据边界

本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-26T07:25:13+00:00 · 数据来源：Paper Collector