Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Introduces least-to-most prompting, combining problem decomposition and step-by-step solving, greatly enhancing large language models' reasoning on complex tasks.
Key Findings
Methodology
The approach employs a two-stage prompting framework: first, demonstrating how to decompose complex problems into simpler subproblems via few-shot examples; second, sequentially solving these subproblems, leveraging previous answers. This method does not require model fine-tuning. It integrates problem decomposition with iterative solution steps, enabling models like GPT-3 code-davinci-002 to generalize beyond exemplars. Experimental results show that this strategy achieves over 99% accuracy on the SCAN benchmark across all splits, vastly outperforming chain-of-thought prompting's 16%. It also improves performance on symbolic and math reasoning tasks, especially in multi-step problems.
Key Results
- On symbolic manipulation tasks, least-to-most achieves 100% accuracy in length generalization, while chain-of-thought drops to 31.8%.
- On the SCAN compositional generalization benchmark, least-to-most reaches 99.7% accuracy with only 14 exemplars, compared to 16% for chain-of-thought.
- In GSM8K and DROP math reasoning datasets, the method boosts accuracy in multi-step problems to 62.4% and 82.4%, respectively, outperforming chain-of-thought significantly.
Significance
This strategy addresses the core challenge of model generalization to more difficult problems without additional training. It demonstrates that problem decomposition and progressive reasoning can unlock higher reasoning capabilities, making models more interpretable and adaptable across diverse tasks. The approach offers a scalable, training-free pathway to enhance AI reasoning, impacting fields from NLP to symbolic AI and education.
Technical Contribution
The paper introduces a novel prompt-based framework inspired by educational psychology, combining problem decomposition with sequential solution steps. It departs from traditional chain-of-thought by explicitly teaching models how to break down problems, enabling high generalization with minimal exemplars. This approach provides a new paradigm for prompt engineering that leverages problem structure without model fine-tuning, opening avenues for more robust, interpretable AI systems.
Novelty
This work is the first to systematically incorporate the concept of progressive problem decomposition into large language model prompting, demonstrating that such a strategy can achieve near-perfect generalization on challenging benchmarks like SCAN with minimal data. It fundamentally differs from prior methods that rely on model architecture changes or extensive training, emphasizing prompt design as a powerful tool for reasoning enhancement.
Limitations
- The method still encounters difficulties with extremely long or deeply nested problems, where decomposition errors accumulate. It heavily depends on prompt quality and exemplar relevance; poor examples can impair performance.
- In tasks requiring deep logical inference or domain-specific knowledge, the approach may need further refinement or hybridization with fine-tuning.
- Computational costs increase with the number of subproblems, and the approach's effectiveness may vary across different models and tasks.
Future Work
Future directions include automating exemplar generation, optimizing prompt structures, and integrating multi-modal inputs for richer problem understanding. Extending the framework to more complex reasoning domains, such as scientific reasoning or multi-hop question answering, is also promising. Combining this prompting strategy with model fine-tuning could further enhance robustness and accuracy.
AI Executive Summary
Despite remarkable advances, large language models still struggle to generalize to complex reasoning tasks that surpass their training exemplars. Traditional chain-of-thought prompting, while effective in simple scenarios, falters when faced with problems requiring solutions beyond the scope of provided demonstrations. Addressing this challenge, the authors propose a novel 'least-to-most' prompting strategy inspired by educational psychology. This method decomposes complex problems into a sequence of simpler subproblems, then solves them step-by-step, leveraging prior answers to facilitate subsequent steps.
The approach involves two key stages: first, demonstrating how to decompose a problem through few-shot examples; second, sequentially solving the subproblems, again guided by few-shot prompts. This process effectively teaches the model to mimic human-like problem-solving, where understanding and solving are broken into manageable parts. The authors validate their method across multiple tasks—symbolic manipulation, compositional generalization (SCAN), and math reasoning (GSM8K, DROP)—using GPT-3 code-davinci-002. Results show a dramatic improvement: on the SCAN benchmark, accuracy reaches 99.7% with only 14 exemplars, far surpassing chain-of-thought's 16%. In symbolic tasks, length generalization hits 100%, and in math reasoning, accuracy increases by over 20 percentage points.
This work signifies a paradigm shift in prompt engineering, emphasizing problem structure and progressive reasoning over mere pattern matching. It demonstrates that models can learn to decompose and conquer complex tasks without additional training, making AI systems more interpretable, flexible, and scalable. The implications extend to education, automated reasoning, and beyond, paving the way for more human-like problem-solving in AI. Future research will explore automating decomposition, integrating multi-modal data, and applying this framework to even more challenging domains, promising a new era of robust, general-purpose reasoning AI.
Deep Dive
Abstract
Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.