OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

核心问题

How can AI agents be systematically evaluated on real-world professional tasks across diverse occupational domains where no existing benchmarks exist?

核心方法

{'approach': 'The authors introduce Language Environment Simulators (LES) that use LLMs to simulate tool-response-level environment interactions for evaluation. The benchmark covers 100 professional task scenarios across 10 industries and 65 specialized domains, with 382 solvable task instances evaluated under clean conditions and three fault injection scenarios (explicit, implicit, and mixed faults).', 'key_components': ['LLMs can serve as world models of the internet for web agent planning, as demonstrated by prior work.', 'WebWorld trains the first open-web simulator at scale on 1M+ interactions for agent training and inference-time search.', 'The LES approach uses LLMs to simulate tool-response-level environment interaction specifically for evaluation purposes.', 'LES supports stateful multi-step professional tasks with realistic action spaces across 100 scenarios and 65 specialized domains.', 'Larger models consistently outperform smaller counterparts within the same family.', 'Performance gaps range from 7.1% to 11.0% between large and small variants.', 'Claude 4.5 is an exception where Opus and Sonnet perform nearly identically.', "The 4.5 generation's architectural improvements benefited both model sizes equally.", 'Each model exhibits a unique pattern of strengths and weaknesses across different industries.', 'Gemini 3.1 Pro excels in knowledge-intensive domains that reward factual accuracy and structured reasoning.'], 'section_ids': ['sec_3', 'sec_21', 'sec_27']}

论点验证

... 共 37 个论点

可复现性评估

较低可复现性 (0%)

局限性（作者自述）

• Language Environment Simulators model domain logic rather than domain data. An LES understands that a drug interaction check should return contraindications, but the specific values it returns are generated rather than retrieved from a real database. This means OCCUBENCH evaluates an agent's decision-making process (whether it checks the right things in the right order) rather than its ability to handle exact real-world data values.
• For domains where precise numerical correctness is critical (e.g., financial calculations to the cent), LES-based evaluation should be complemented with real-environment testing.
• As our cross-simulator experiments demonstrate, evaluation results are tied to the specific simulator used during data synthesis. Tasks verified as solvable under Gemini-3-Flash may become unsolvable under a different LES, and agent rankings can shift when the simulator changes. This is an inherent limitation of any LES-based evaluation: the simulator is part of the evaluation apparatus, not a neutral observer.