Using CascadeBench (27,243 transitions, 64 worlds), the paper shows fine-tuned models achieve 82-91 IoU in-distribution but collapse to ~40-41 under configuration shift.
论文原始摘要
World models enable agents to anticipate the effects of their actions by internalizing environment dynamics. In enterprise systems, however, these dynamics are often defined by tenant-specific business logic that varies across deployments and evolves over time, making models trained on historical transitions brittle under deployment shift. We ask a question the world-models literature has not addressed: when the rules can be read at inference time, does an agent still need to learn them? We argue, and demonstrate empirically, that in settings where transition dynamics are configurable and readable, runtime discovery complements offline training by grounding predictions in the active system instance. We propose enterprise discovery agents, which recover relevant transition dynamics at runtime by reading the system's configuration rather than relying solely on internalized representations. We introduce CascadeBench, a reasoning-focused benchmark for enterprise cascade prediction that adopts the evaluation methodology of World of Workflows on diverse synthetic environments, and use it together with deployment-shift evaluation to show that offline-trained world models can perform well in-distribution but degrade as dynamics change, whereas discovery-based agents are more robust under shift by grounding their predictions in the current instance. Our findings suggest that, in configurable enterprise environments, agents should not rely solely on fixed internalized dynamics, but should incorporate mechanisms for discovering relevant transition logic at runtime.
Paper Collector 中文速览
企业系统中运行时发现配置比学习动态更鲁棒
方法概述
设计企业发现代理，在推理时动态读取系统配置而非仅依赖离线训练；构建CascadeBench基准测试，采用World of Workflows评估方法论；在合成环境中对比离线训练世界模型与发现代理在部署偏移下的性能
核心贡献
提出企业发现代理，通过运行时读取系统配置而非依赖内部化表示来恢复动态，引入CascadeBench基准测试，证明在部署偏移下更鲁棒
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
Should LLM agents in enterprise systems rely on learned world models or discover dynamics at runtime through interaction with the system?
核心方法
The authors introduce CascadeBench with 27,243 verified transition samples across 64 worlds spanning 6 industries. They compare three approaches: prompted baselines, learned world models fine-tuned on transition tuples, and enterprise discovery agents that retrieve configuration at inference time. Evaluation uses IoU metrics across three transition complexity tiers (schema-determined, rule-composed, execution-inferred).
方法组件
- Three approaches are compared: prompted baseline, learned world model, and enterprise discovery agent.
- All approaches take current state s_t and proposed action a_t as input.
- All approaches output structured field-level diffs describing the predicted transition.
- Predicts transitions by internalizing dynamics from supervised training data.
- Fine-tuned on (s_t, a_t, s_{t+1}) tuples from Enterprise Gym.
- Training target is the minimal field-level diff between current and next state.
- Tests whether learned dynamics transfer to instances with different industries, structures, and rule sets.
论点验证
The definition is explicitly provided in p_3, and the implementation in Appendix G (p_75-89) demonstrates how this is realized through the snow_query tool that queries business rules, record states, choice values, and SLA definitions.
The three tiers are clearly defined in p_17 with concrete examples provided in Table 5 (p_54-61). Tier-stratified results are reported throughout the paper and in Appendix C.
The formal definition W = (E, T) is explicitly provided in p_18 with clear specification of what E contains and what T represents.
Specific numbers are provided in p_19: 1,596 business rule patterns, 6 industries, 11 operational domains, ~27,000 LLM-generated base scenarios, ~802,000 validated initial states. The construction pipeline is detailed in Appendix E.
Specific numbers are provided in p_20: 27,243 verified transition samples, 64 worlds, 6 industries (financial services, government, healthcare, manufacturing, retail, technology), 3 organizational sizes.
Stated in p_22 and the mechanism is explained in p_68: reserved namespace prefixes used by ServiceNow's product modules are excluded so schemas cannot overlap with platform tables seen during pretraining.
Stated in p_22 as a key design feature. The evaluation settings in p_34 (w/ BR and w/o BR) demonstrate this controllability in practice.
Stated in p_22 and the filtering process is described in p_71: internal metadata fields are filtered out, retaining only semantically meaningful content changes.
Specific IoU ranges are provided in p_36, referencing Table 1: 9-16 IoU(T+F) for CascadeBench w/o BR vs 21-23 for WoW. This is self-reported experimental data.
Specific improvement numbers are provided in p_36: ~2-3 IoU points on CascadeBench w/o BR and ~10 points on WoW. This is self-reported experimental data.
Specific IoU numbers are provided in p_37, referencing Figure 3: 91.6 for Gemma-4-31B and 82.0 for Qwen-3.6-27B in-distribution, dropping to 40-41 on CascadeBench. This is self-reported experimental data.
Specific IoU improvements are provided in p_39, referencing Figure 4: Opus 4.6 rises from 0.40 to 0.45, Sonnet 4.6 from 0.32 to 0.44 at k=1. This is self-reported experimental data.
Specific IoU ranges are provided in p_41, referencing Table 2: ~10 IoU for no-BR baseline vs mid-20s to low-30s for discovery agent. This is self-reported experimental data.
Specific IoU numbers are provided in p_42, referencing Figure 5: 0.56-0.58 for Tier 1, with near-collapse on Tier 2 and Tier 3. This is self-reported experimental data.
Stated in p_42 with reference to Figure 5, showing discovery stays near oracle on Tier 1 and Tier 2 while outperforming prompted baseline. This is self-reported experimental data.
Stated in p_44 with reference to Figure 4, showing discovery agents outperform matched prompted baselines across rollout horizons with performance decreasing as k grows. This is self-reported experimental data.
This is an interpretation supported by the experimental comparison between w/ BR and w/o BR conditions. The large performance gap when rules are provided vs not provided demonstrates that rules carry the dynamics being measured.
... 共 52 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- Fine-tuning hyperparameters: learning rate, batch size, number of epochs, LoRA rank/alpha/dropout parameters, optimizer settings
- Training data details: size of Enterprise Gym dataset, number of (s_t, a_t, s_t+1) tuples, train/val/test splits
- Random seeds for reproducibility of training and evaluation
- Hardware specifications: GPU types, memory requirements, training time
- Exact prompt templates for prompting baseline and discovery agent system prompts
- Full implementation of snow_query tool specification and discovery agent architecture
- CascadeBench and WoW benchmark construction details: number of examples, schema specifications, configuration parameters
- Exact implementation of IoU(T+F) and Strict IoU metrics for field-level diff evaluation
- State representation format: how enterprise states are serialized, action encoding, diff computation method
- Statistical details: number of experimental runs, confidence intervals, significance tests
局限与证据边界
- The discovery agent assumes business rules are readable on the live instance; access controls degenerate it to the prompted baseline.
- DA performance also depends on tool-use capability: on open-weight models in the 27-31B range, the retrieval loop is unreliable enough that LoRA finetuning wins in some conditions, so the choice between training and discovery is deployment-dependent.
- Our evaluation is single-platform (ServiceNow) and our quantitative results focus on Tier 1 and Tier 2 transitions; Tier 3 results are reported in Appendix C.
- Tier 3 stratification is limited to multi-rule conflicts detectable from the audit log, and broader execution-order dynamics remain out of scope.
- The DA-vs-trained comparison rests on a small set of open-weight models where LoRA finetuning is feasible.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-17T07:17:24+00:00 · 数据来源：Paper Collector