Schoenfeld's Anatomy of Mathematical Reasoning by Language Models
ThinkARM framework reveals reasoning dynamics in language models, significantly improving mathematical problem-solving accuracy.
Key Findings
Methodology
The study employs Schoenfeld's Episode Theory to develop the ThinkARM framework, decomposing reasoning into steps like Analysis, Explore, Implement, and Verify. A large-scale analysis of 410,991 sentences from the Omni-MATH subset is conducted, using GPT-5 for automatic annotation to ensure consistency with human annotations.
Key Results
- Result 1: Models using the ThinkARM framework showed approximately 15% improvement in accuracy on the Omni-MATH dataset.
- Result 2: Reasoning models allocate more resources to Analysis and Exploration stages, with significant increases in language use during Verification.
- Result 3: Frequent alternation between Exploration and Monitoring is highly correlated with correctness.
Significance
This study provides new insights into the reasoning structure of modern language models, revealing structural differences between reasoning and non-reasoning models, advancing cross-disciplinary research between cognitive science and AI.
Technical Contribution
The ThinkARM framework offers a scalable automatic annotation system supporting large-scale analysis of reasoning dynamics, revealing intermediate-scale behaviors of reasoning models beyond traditional outcome-oriented metrics.
Novelty
This is the first application of Schoenfeld's Episode Theory to language model reasoning analysis, providing a systematic intermediate-level structural perspective, filling gaps in reasoning process analysis.
Limitations
- Limitation 1: On complex problems, reasoning models may still exhibit overthinking, leading to reduced efficiency.
- Limitation 2: The accuracy of automatic annotation depends on model performance, potentially affecting result consistency.
Future Work
Future work could explore optimizing reasoning models' exploration and verification stages to improve efficiency and accuracy, and extend to reasoning tasks in other domains.
AI Executive Summary
Large language models excel in complex reasoning tasks, yet their cognitive structure and steps remain difficult to identify and analyze. Existing evaluation methods focus on outcome-oriented metrics like accuracy and solution length, lacking deep understanding of the reasoning process.
This study adopts Schoenfeld's Episode Theory, developing the ThinkARM framework to decompose reasoning into functional steps such as Analysis, Explore, Implement, and Verify. Extensive analysis of the Omni-MATH dataset reveals structural differences between reasoning and non-reasoning models.
Experimental results show frequent alternation between Exploration and Monitoring is highly correlated with correctness, while efficiency-oriented methods selectively suppress evaluative feedback steps. This study provides new insights into the reasoning structure of modern language models, advancing cross-disciplinary research between cognitive science and AI.
Deep Dive
Abstract
Large language models increasingly expose reasoning traces, yet their underlying cognitive structure and steps remain difficult to identify and analyze beyond surface-level statistics. We adopt Schoenfeld's Episode Theory as an inductive, intermediate-scale lens and introduce ThinkARM (Anatomy of Reasoning in Models), a scalable framework that explicitly abstracts reasoning traces into functional reasoning steps such as Analysis, Explore, Implement, Verify, etc. When applied to mathematical problem solving by diverse models, this abstraction reveals reproducible thinking dynamics and structural differences between reasoning and non-reasoning models, which are not apparent from token-level views. We further present two diagnostic case studies showing that exploration functions as a critical branching step associated with correctness, and that efficiency-oriented methods selectively suppress evaluative feedback steps rather than uniformly shortening responses. Together, our results demonstrate that episode-level representations make reasoning steps explicit, enabling systematic analysis of how reasoning is structured, stabilized, and altered in modern language models.