LIMO: Less is More for Reasoning
LIMO achieves complex reasoning with minimal data, scoring 63.3% on AIME24 and 95.6% on MATH500.
Key Findings
Methodology
LIMO uses 800 high-quality samples for supervised fine-tuning to elicit complex reasoning. Data curation involves multi-stage filtering for question difficulty and reasoning chain quality. The model is based on Qwen2.5-32B-Instruct, optimized with DeepSpeed ZeRO-3 and FlashAttention-2.
Key Results
- Result 1: Achieved 63.3% accuracy on AIME24, significantly outperforming prior fine-tuned models (6.5%).
- Result 2: Scored 95.6% on MATH500, surpassing QwQ-32B-Preview's 89.8% by 5.8 percentage points.
- Result 3: Demonstrated strong out-of-distribution generalization, achieving 84.2% accuracy on CHMath, a 45.8% improvement over baselines.
Significance
This study demonstrates that complex reasoning can be elicited with minimal yet high-quality data, reducing computational costs and addressing data efficiency challenges. The findings have significant implications for both academia and industry.
Technical Contribution
Introduced the LIMO hypothesis, emphasizing the importance of pre-trained knowledge completeness and high-quality training samples. Developed a meticulous data curation pipeline and efficient fine-tuning methodology to enhance reasoning performance.
Novelty
First to systematically validate the 'Less-Is-More' hypothesis for reasoning tasks, proposing a minimal data training approach that drastically reduces data requirements.
Limitations
- Limitation 1: Relies heavily on the completeness of pre-trained knowledge; weaker pre-trained models may underperform.
- Limitation 2: Data curation is complex and time-consuming, posing challenges for smaller research teams.
- Limitation 3: Current experiments focus on mathematical reasoning; applicability to other domains remains untested.
Future Work
Future studies could explore extending LIMO to multi-domain reasoning tasks, optimizing data curation pipelines for scalability, and improving cross-lingual and cross-distribution generalization.
AI Executive Summary
Complex reasoning has long been a challenge for large language models, traditionally requiring vast amounts of training data and incurring high computational costs. The LIMO model introduces a novel approach by leveraging minimal but high-quality data to elicit reasoning capabilities. Using only 800 carefully curated samples, LIMO fine-tunes the Qwen2.5-32B-Instruct model and achieves remarkable accuracy: 63.3% on AIME24 and 95.6% on MATH500, outperforming models trained on orders of magnitude more data.
Experimental results show that LIMO excels not only on in-domain benchmarks but also in out-of-distribution scenarios, such as achieving 84.2% accuracy on CHMath, a 45.8% improvement over baselines. This validates the 'Less-Is-More' hypothesis, proving that high-quality examples can effectively activate complex reasoning in pre-trained models.
While LIMO demonstrates outstanding performance in mathematical reasoning, its reliance on pre-trained knowledge completeness and the complexity of data curation are notable limitations. Future research could focus on expanding its applicability to diverse domains, streamlining data selection processes, and enhancing cross-lingual and cross-distribution generalization.
Deep Dive
Abstract
We challenge the prevailing assumption that complex reasoning in large language models (LLMs) necessitates massive training data. We demonstrate that sophisticated mathematical reasoning can emerge with only a few examples. Specifically, through simple supervised fine-tuning, our model, LIMO, achieves 63.3\% accuracy on AIME24 and 95.6\% on MATH500, surpassing previous fine-tuned models (6.5\% on AIME24, 59.2\% on MATH500) while using only 1\% of the training data required by prior approaches. Furthermore, LIMO exhibits strong out-of-distribution generalization, achieving a 45.8\% absolute improvement across diverse benchmarks, outperforming models trained on 100x more data. Synthesizing these findings, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning can emerge through minimal but strategically designed demonstrations of cognitive processes. This hypothesis suggests that the threshold for eliciting complex reasoning is not dictated by task complexity but rather by two key factors: (1) the completeness of the model's pre-trained knowledge base and (2) the effectiveness of post-training examples in serving as "cognitive templates" that guide reasoning.