π-BENCH introduces a benchmark with 100 multi-turn tasks across 5 personas for evaluating proactive personal assistants in long-horizon workflows. Results show significant gaps between task completion and proactivity, with history being crucial for proactive behavior.
论文原始摘要
The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $π$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $π$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.
Paper Collector 中文速览
评估个人助手在长期工作流中的主动能力
方法概述
构建包含隐藏用户意图、任务间依赖和跨会话连续性的评测数据集，通过多轮交互同时测量智能体的主动性和任务完成度，在长期轨迹中评估真实场景表现
核心贡献
提出π-Bench基准，包含100个多轮任务和5种用户画像，评估智能体识别隐藏意图和跨会话连续性的主动能力
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
How can we evaluate proactive personal assistant agents that must infer hidden intents from underspecified requests in long-horizon workflows with persistent workspaces and cross-session dependencies?
核心方法
The authors construct π-BENCH with 100 multi-turn tasks across five domain-specific user personas, each episode containing 20 sessions with cross-session dependencies. Agents are evaluated using two metrics: proactivity (PROC) measuring fraction of hidden intents resolved through direct completion or targeted elicitation, and completeness (COMP) measuring satisfaction of verifiable task requirements via checklist-based evaluation. Nine frontier models are tested under a unified ReAct-style agent scaffold with tool interfaces.
论点验证
This claim combines the benchmark introduction with its structural details. Both aspects are well-documented with specific numbers (100 tasks, 5 personas, cross-session dependencies) and formal evaluation metrics (PROC and COMP).
The five user roles are explicitly listed: researcher, marketer, law trainee, pharmacist, and financier. This is a concrete design specification that is clearly stated and verifiable.
The paper explicitly states: 'Each episode contains 20 sessions, and each session corresponds to one task addressed through multi-turn interaction.' This is a concrete, verifiable design specification.
The paper provides formal definitions for both metrics: Proactivity (PROC) in Section 3.5 with equation, and Completeness (COMP) in Section 3.5 with equation. Both are mathematically defined and operationally specified.
The paper describes the agent paradigm focusing on long-horizon personal agents that produce concrete artifacts. This is a design choice that is clearly stated, though it's more of a scope definition than a testable claim.
The paper mentions grounding in practical tool and skill interfaces with examples (shopping tool, web search tool, data processing skill). The claim is stated but specific tool implementations are not exhaustively detailed.
The paper states roles are constructed with domain experts and normalized. This is a design choice claim that is stated but the expert involvement is not quantified or detailed with specific methodology.
The paper describes task derivation from domain experts' authentic work routines. This is stated as part of the methodology, though the specific expert involvement process is not exhaustively documented.
The paper provides specific numbers: six strong dependency groups (2-3 tasks each) and five largely independent tasks. These are concrete, verifiable design specifications.
The paper provides formal definition of checklists with notation C = {c_1, ..., c_n} and describes their role in defining verifiable completion criteria. This is a concrete design specification.
The paper explicitly describes two types of graders: rubric-based evaluation for open-ended content and rule-based verification for objective conditions. Both are described with implementation details.
The paper provides the formal mathematical definition: PROC = (|I_com| + |I_inf|) / |I|. This is a precise, verifiable definition.
The paper states all models are evaluated under the same agentic scaffold adapted from Nanobot. This is a methodological choice that controls for scaffold variation. The reference to Nanobot [11] is provided.
The paper explicitly states: 'we run each task three times with independent trajectories and report averaged results with standard deviations.' This is a concrete methodological specification.
The paper explicitly states: 'We use GPT-5.4 as the base model for the user agent and as the rubric-based grader, with temperature set to zero.' This is a concrete, verifiable specification.
The paper provides exact numbers: 84.1% COMP vs. 38.1% PROC for legal matter operations and handoffs (H). These are specific quantitative results from the experiments.
... 共 38 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- No code repository available - the benchmark implementation, agent scaffold, and evaluation scripts are not publicly released
- No benchmark data available - the actual tasks, hidden intents, and rubrics defining π-BENCH are not accessible
- Model hyperparameters not specified - temperature, max tokens, number of turns/iterations, timeout settings for all evaluated models
- Number of repeated runs per task not specified (only mentioned as 'repeated runs' without exact count)
- Random seeds not reported for reproducibility of stochastic model outputs
- Exact prompts used for agent scaffold and evaluation judges (GPT-5.4 rubric grading and status assignment) not provided
- Specific modifications made to Nanobot scaffold not detailed
- Hidden intent definitions and categorization scheme not fully specified
- Checklist rubric details for COMP scoring not provided
- Exact model API versions/release dates not specified (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro)
局限与证据边界
- The users are simulated rather than real humans, which is difficult to avoid because long horizon evaluation with live users is costly, hard to reproduce, and difficult to scale.
- Our experiments use a single agentic scaffold adapted from Nanobot, which provides a controlled evaluation setup but may not capture the full variation introduced by alternative scaffolds, which can add substantial adaptation effort and scaffold-specific confounds.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-24T07:19:42+00:00 · 数据来源：Paper Collector