SOOHAK presents a mathematician-curated benchmark evaluating LLMs on research-level mathematics through 340 Challenge and 99 Refusal items. Top models achieve only 30.39% accuracy, with open-weight systems at 13.87%, exposing substantial limitations in advanced mathematical reasoning.
论文原始摘要
Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.
Paper Collector 中文速览
数学家创建439题研究级数学基准，评估LLM推理与拒绝能力
方法概述
组织64位数学师从零创作439道研究级数学问题，分挑战集测试推理能力、拒绝集测试识别不当问题能力。通过Gemini-3-Pro、GPT-5等前沿模型和开源模型评测，验证基准难度与区分度。2026年底公开以防污染。
核心贡献
发布由64位数学家全新创作的439题研究级数学基准Soohak，含挑战集和拒绝集，填补研究级数学评测空白
原始来源与核验范围
核验仅覆盖论文正文中的主张、证据片段和参考文献关系。NGJOO 未独立运行作者代码、重做实验或验证真实部署效果；页面中的可复现性分数是文档完整度评估，不是复现实验结果。
核心问题
How do large language models perform on research-level and graduate-level mathematical problems, and can they appropriately identify and refuse ill-posed questions?
核心方法
The benchmark was constructed by 105 mathematicians and students who authored original problems across three splits: Challenge (340 research-level items), Refusal (99 ill-posed items), and SOOHAK-Mini (702 contest-level items). Problems passed through model-gated collection gates requiring failure of progressively larger baseline models. Eleven LLMs were evaluated with three independent responses per question, using GPT-5-Mini as an LLM judge for answer equivalence.
方法组件
- Eleven language models evaluated spanning closed and open-weight systems.
- Closed systems: Gemini-3-Pro, Gemini-3-Flash, GPT-5 Medium, Claude-Opus-4.5, Claude-Sonnet-4.5, Grok-4.1-Fast.
- Open-weight systems: Qwen3-235B-A22B-thinking-2507, GPT-OSS-120B, Kimi-2.5, GLM-5.
- Reasoning was enabled for all models during evaluation.
- Three independent responses sampled per model-question pair, reporting avg@3 and pass@3.
论点验证
The paper states SOOHAK-Mini is a 702-question collection by 105 contributors (p_4, p_10). The contributor breakdown is detailed in p_10 (86 primary-system + 19 ScienceBench = 105) and p_43, providing strong evidence for these specific numbers.
The paper states 25 participants across five teams (p_5) and provides detailed team descriptions with varying expertise profiles (p_27, p_36). The human evaluation setup is thoroughly described in §6.
Specific numerical result from Table 2. Kimi-2.5 at 13.87% Avg@3 on Challenge is reported as the best open-weight model. Self-reported result, confidence capped at 0.85.
Specific numerical results from Table 2 for the Refusal subset. The best closed model (Gemini-3-Flash at 43.10%) and GLM-5 (49.49%) are reported with exact numbers. Self-reported, confidence capped at 0.85.
Specific numerical results from Table 2 for SOOHAK-Mini. GPT-5 at 72.22% and Kimi-2.5 at 66.07% are reported. Self-reported, confidence capped at 0.85.
The 50.6% coverage figure is reported for the aggregated human teams on 79 prompts (p_5). The evaluation setup is described in detail (p_28: 49 Calibration + 30 Challenge = 79 prompts). Self-reported human evaluation result, confidence capped at 0.85
Submission requirements are clearly specified in p_10: English or Korean, text-only LaTeX, complete solution, explicit final-answer line. These are concrete, verifiable format requirements.
Specific financial details are provided in p_10: USD 260,000 total pool, per-accepted-question payments, split-dependent payments ranging USD 36-3,623, cap of USD 20,000 per contributor. These are concrete, verifiable numbers.
The sampling methodology is clearly specified in p_22: three independent responses per model-question pair, reporting avg@3 and pass@3. The metric formulas are provided.
Human evaluation conditions are clearly specified in p_29: nominal 4.5 hour time budget, permitted non-AI tools (programming environments, computer algebra systems, internet search), prohibited LLMs and AI-assisted search.
The scoring methodology is clearly specified in p_30: purely outcome-based, correct only if final answer is correct, no partial credit.
Specific numerical results reported in p_24: 124 Challenge items unsolved by any model, 170 total unsolved or missed. These are derived from the evaluation results in Table 2. Self-reported, confidence capped at 0.85.
... 共 40 个论点
可复现性评估
较低可复现性 (0%)
缺失的复现细节
- The actual benchmark questions/prompts (79 evaluation items) are not provided in the paper
- Specific decoding parameters for each model (temperature, top-p, etc.) - referenced in Appendix D.1 but not accessible
- Random seeds used for LLM evaluations
- Exact prompt templates/formatting used when querying LLMs
- Hardware specifications and API versions/dates for closed models
- Grading methodology - how correctness was determined and by whom
- Number of human participants and their qualifications/background
- Sampling methodology for selecting the 79 prompts from the full dataset
- Statistical methods for computing Avg@3 and Pass@3 metrics
- The novamath.github.io link mentioned does not provide accessible benchmark data
局限与证据边界
- SOOHAK and SOOHAK-Mini were assembled under unusual constraints, with a large budget of roughly USD 550,000 or 800M KRW, but only four months for recruitment, collection, review, and human baseline studies.
- The main lesson is that difficulty labels are noisy and incomplete proxies for benchmark value.
- In retrospect, this compressed timeline was a first-order bottleneck. It limited how early we could establish reviewer infrastructure, how broadly we could recruit across subfields, and how many iterative pilot rounds we could run to identify failure modes in question quality before scaling.
- Because the project initially prioritized domestic recruitment, we encountered an unavoidable limitation. Even with excellent local experts, a geographically constrained pool reduces the breadth of subfields and styles represented.
- The "unique integer answer" format is becoming exhausted. This format is attractive because it enables automatic scoring and reduces ambiguity, and it makes random guessing unlikely when the answer space is large. However, it also restricts the space of feasible problems and systematically favors subfields where clean numeric end-answers are natural.
本分析由 PDF 阅读助手 自动生成，仅供参考，不构成学术评审意见。验证结论和可复现性评估基于论文文本自动分析，可能存在偏差。原始论文请参阅 arXiv。
分析时间：2026-05-19T07:11:22+00:00 · 数据来源：Paper Collector