LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling - AI 论文深度分析

TL;DR
AutoTTS introduces an environment-driven framework that automatically discovers test-time scaling strategies for LLMs through offline replay environments and beta parameterization. The discovered controller reduces token consumption by ~69.

论文原始摘要

Test-time scaling (TTS) has become an effective approach for improving large language model performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework, AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width--depth TTS as controller synthesis over pre-collected reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the agent diagnose why a TTS program fails. Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy--cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes. Our data, and code will be open-source at https://github.com/zhengkid/AutoTTS.

Paper Collector 中文速览

AutoTTS自动发现测试时扩展策略，优化准确率-成本权衡

方法概述

构建轨迹和探针信号环境，将TTS建模为控制器合成问题。控制器决策分支、继续、探针、剪枝或停止。通过β参数化使搜索可处理，细粒度执行轨迹反馈提高发现效率。

核心贡献

提出环境驱动框架AutoTTS，将TTS策略发现自动化，在数学推理任务上超越人工基线，发现成本仅$39.9

原始来源与核验范围

核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。

已证实

证据不足

无法验证

N/A

可复现性

置信度

78%

核心问题

Can test-time scaling strategies for large language models be automatically discovered through environment-driven search rather than manual hand-crafting of heuristics?

核心方法

AutoTTS constructs offline replay environments by pre-collecting reasoning trajectories, enabling cheap controller evaluation without repeated LLM calls. The framework uses beta parameterization (single scalar parameter) to prevent overfitting and execution trace feedback to diagnose failure modes. A multi-round discovery loop with Claude Code as the explorer agent searches for effective allocation policies.

方法组件

The discovered controller with β = 1 achieves best accuracy on DeepSeek-R1-Distill-Llama-8B with HMMT25 while reducing tokens from 985.7K to 533.9K compared to SC@64.
On GPQA-Diamond, the β = 0.5 variant reduces token cost from 510.0K to 151.0K while maintaining comparable accuracy to SC@64.
The discovered strategies transfer successfully to both a different model family (Llama-based) and a non-math reasoning benchmark.
Results suggest the discovered controllers are not tightly overfitted to the main Qwen-based math setting.

论点验证

已证实 (85%) In this work, we propose AUTOTTS, an environment-driven paradigm for automatic TTS strategy discovery (Figure 1). Instead of hand-crafting individual branching, pruning, and stopping heuristics, AutoTTS shifts the human role to constructing discovery environments, where humans define the control space through states, actions, feedback, and objectives, and agents search within this space for effective allocation policies.

已证实 (85%) As a proof-of-concept instantiation, we formulate width-depth test-time scaling (Figure 2) as controller synthesis in an offline replay environment. For each problem, we pre-collect reasoning trajectories and intermediate probe signals, allowing a controller to replay decisions over when to branch, continue, probe, prune, or stop.

已证实 (80%) To make controller search tractable, we introduce beta parameterization, where each controller exposes only one scalar trade-off parameter β and derives all internal hyperparameters deterministically from it, reducing overfitting to the search set.

已证实 (80%) Additionally, we address the feedback issue by logging execution traces that expose how each controller allocates computation over time, enabling the explorer to diagnose failure modes and propose targeted improvements.
Execution traces are described in p_22 and validated through ablation in Table 3. The ablation shows that without execution traces, the discovered controller achieves worse performance while allocating more tokens, confirming their importance.

已证实 (85%) We construct an offline replay environment that moves all LLM calls prior to the discovery process, making controller evaluation deterministic and cheap.
The offline replay environment is fully described in p_17-p_19, with concrete implementation details: pre-collecting N trajectories, segmenting into fixed-length intervals, and deterministic replay. The cost benefit is quantified in p_36 ($39.9, 160

已证实 (80%) The discovered controller, which we term the Confidence Momentum Controller (CMC), reveals four non-obvious mechanisms: trend-based stopping via EMA momentum, coupled width-depth control through a shared evidence signal, alignment-aware depth allocation, and conservative branch abandonment.

已证实 (75%) The discovered controller, optimized solely on AIME24, generalizes to held-out benchmarks AIME25 and HMMT25, outperforming all handcrafted baselines in three out of four models on average, and remaining competitive on Qwen3-8B (62.7 vs. 62.8 for SC@64).

已证实 (75%) When β = 0.5, it reduces token consumption by approximately 69.5% compared to SC@64 while maintaining on-par accuracy on average across all four models (45.3 vs. 45.2); when β = 1.0, it pushes peak accuracy beyond all handcrafted baselines in 5 out of 8 cases.

已证实 (85%) Overall, the five-round discovery process costs only $39.9 and takes 160 minutes, suggesting that our replay-based discovery framework is practical to run.
Specific cost numbers are provided in p_36: $39.9 and 160 minutes for five-round discovery. These are concrete, verifiable metrics that directly support the claim of practical affordability.

已证实 (80%) On DeepSeek-R1-Distill-Llama-8B with HMMT25, our discovered controller with β = 1 achieves the best accuracy among all methods while also reducing total tokens from 985.7K under SC@64 to 533.9K.
Table 2 and p_32 provide specific numbers: best accuracy among all methods, token reduction from 985.7K to 533.9K. The quantitative data directly supports the claim.

已证实 (80%) On GPQA-Diamond, the β = 0.5 variant reduces the token cost from 510.0K to 151.0K, and also improves over ASC in both accuracy and token usage.
p_32 provides specific numbers: token cost reduction from 510.0K to 151.0K, improvement over ASC in both accuracy and token usage. Quantitative data directly supports the claim.

已证实 (80%) Removing Beta Parameterization leads to controllers with excessive free hyperparameters, creating a large search space that makes it easy to overfit to E search. This overfitting manifests as overly aggressive pruning and stopping thresholds that happen to work well on the search set but fail to generalize, resulting in a drastic token reduction (575.5K to 93.3K) accompanied by an accuracy drop from 53.1 to 49.0 on held-out benchmarks.
Table 3 and p_34-p_35 provide specific ablation numbers: token reduction from 575.5K to 93.3K, accuracy drop from 53.1 to 49.0. The quantitative ablation data directly supports the claim about overfitting.

已证实 (75%) Without execution traces, the discovered controller achieves worse performance while allocating much more tokens. This indicates that final acc/token numbers alone are insufficient to guide effective search.

已证实 (80%) The discovered controller consistently achieves a stronger accuracy-efficiency frontier, as indicated by its corresponding curves being positioned above the other curves across all four settings.
Figure 3 visualizes the accuracy-token scaling curves, and p_30 describes the curves being positioned above baselines across all four settings. The visual evidence in Figure 3 directly supports the claim.

已证实 (85%) Following the data collection protocol of Parallel-Probe [6], for each question q ∈ Q, we pre-collect N independent reasoning trajectories from the base LLM, each segmented into fixed-length intervals of ∆ tokens.
The design choice is fully described in p_18, following Parallel-Probe protocol. This is a methodological specification that is implemented and used throughout experiments.

已证实 (85%) We pre-sample 128 reasoning trajectories per (model, problem) pair at temperature 0.7 with a probing interval of 500 tokens to construct the replay matrix.
Specific hyperparameters are provided in p_26: 128 trajectories, temperature 0.7, 500-token probing interval. These are concrete, verifiable design choices.

已证实 (85%) To reduce variance, each controller is evaluated 64 times independently by randomly sampling a subset of trajectories from the pre-sampled pool of 128, and the results are averaged.
p_26 explicitly states 64 independent evaluations with random sampling from the 128-trajectory pool. This is a concrete, verifiable design choice for variance reduction.

已证实 (85%) For discovery, we use AIME24 as Q search and construct E search as the union of AIME24 environments across all four models. The discovery loop runs for five rounds with Claude Code as the agent, and the final controller is selected as the one achieving the highest accuracy on E search.
p_26 provides complete specification: AIME24 as Q_search, union across four models as E_search, five rounds, Claude Code as agent, selection by highest accuracy.

已证实 (85%) We partition Q into a search set Q search and a held-out evaluation set Q eval.
p_20 explicitly states the partition into Q_search and Q_eval. This is a standard methodological choice that is clearly described.

已证实 (85%) We further require this map to be monotone, such that larger β corresponds to larger token budget. This collapses the search space to a one-dimensional sweep and prevents the agent from discovering sharp, search-set-specific thresholds.
p_25 describes the monotonicity requirement and its purpose: collapsing search space to 1D sweep, preventing search-set-specific thresholds. This is a well-motivated design choice.

... 共 38 个论点

可复现性评估

较低可复现性 (0%)

缺失的复现细节

No code repository available - core implementation details of the discovery framework and discovered controller are inaccessible
No data/benchmark links provided - exact versions of AIME24, AIME25, HMMT25, GPQA-Diamond datasets used are not specified
Discovered controller details referenced in 'Appendix D' but not included in provided text - the key contribution is not documented
Random seeds not specified for trajectory sampling (128 trajectories) or controller evaluation (64 independent runs)
Claude Code agent configuration missing - version, API settings, prompts/templates used for discovery loop
Hardware specifications not reported - GPU types, memory, compute resources used for experiments
Software environment not documented - Python version, key libraries, framework versions
Discovery loop implementation details missing - what happens in each of the 5 rounds, specific prompts and procedures
Beta parameterization details incomplete - how β values control the tradeoff is not fully explained
Execution traces details missing - mentioned in ablation study but implementation not described

局限与证据边界

The current instantiation constructs environments for width-depth TTS control. It would be interesting to extend the action set and build richer environments that support more complex control structures.
The discovery process currently relies on a frontier coding agent; exploring whether open-source coding agents can achieve comparable discovery performance is an interesting direction for future work.
In our preliminary experiments, we empirically find that agents tend to propose TTS controllers with a large number of hyperparameters, up to 10. With only five discovery rounds, navigating this high-dimensional space causes the agent to collapse onto extreme solutions-such as overly aggressive pruning thresholds-that happen to minimize token cost on the search set but fail to represent robust allocation strategies.

本分析由 PDF 阅读助手自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。

分析时间：2026-05-17T01:13:23+00:00 · 数据来源：Paper Collector