CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning
CORE algorithm enables rapid reasoning improvement using fewer samples and rollouts through contrastive reflection.
Key Findings
Methodology
CORE is a non-parametric learning algorithm that generates insights by contrasting successful and unsuccessful reasoning traces. These insights help the model apply strategies and constraints more effectively in future problems.
Key Results
- CORE outperforms baseline methods like GRPO and GEPA across four reasoning tasks, achieving over 50% improvement in accuracy with fewer rollouts.
- Under fixed rollout budgets, CORE achieves the strongest performance in most task-data regimes.
- CORE insights are interpretable learning artifacts expressed in natural language, reducing risks associated with opaque parameter updates.
Significance
CORE provides a more efficient and interpretable model self-improvement route by extracting contrasts between successful and unsuccessful reasoning traces, addressing long-standing inefficiencies in learning from verifiable rewards.
Technical Contribution
CORE fundamentally differs from existing methods by storing transparent natural-language insights instead of weight updates or single rollouts, offering new engineering possibilities for frozen model self-improvement.
Novelty
CORE is the first to use contrastive reflection for language model self-improvement, differing from reward-based learning methods by achieving more efficient learning through natural language insights.
Limitations
- CORE still requires a substantial number of initial samples to estimate baseline accuracy for some tasks.
- Without similar successful cases, CORE may struggle to generate useful insights.
Future Work
Future work could explore applying CORE to more tasks and improving the quality and efficiency of insight generation.
AI Executive Summary
The CORE algorithm achieves rapid reasoning improvement for language models through contrastive reflection. Existing parametric and non-parametric methods typically require extensive samples and rollouts, whereas CORE reduces these demands by generating natural language insights.
CORE outperforms baseline methods across four reasoning tasks, achieving higher accuracy with fewer rollouts. It generates insights by contrasting successful and unsuccessful reasoning traces and effectively applies them to future problems.
CORE insights are interpretable natural language learning artifacts, reducing risks associated with opaque parameter updates. This suggests CORE offers a more efficient and interpretable model self-improvement route than existing methods.
Deep Analysis
Background
Language models often require significant data and computational resources to learn from verifiable rewards. Existing methods like GRPO and GEPA have efficiency limitations, while CORE offers a more efficient learning route through contrastive reflection.
Core Problem
Existing reasoning task learning methods require extensive samples and rollouts, making them costly and challenging to implement. CORE aims to reduce these demands through contrastive reflection.
Innovation
CORE generates natural language insights by contrasting successful and unsuccessful reasoning traces. These insights help the model apply strategies and constraints more effectively, differing from existing methods.
Methodology
- �� CORE builds two external memory stores: insight memory and rollout memory.
- �� After each training failure, CORE generates insights by contrasting successful and unsuccessful rollouts.
- �� Insights are retrieved based on semantic similarity and utility estimates.
Experiments
CORE is evaluated across four reasoning tasks, including Tower of Hanoi and MathGAP. Comparisons are made using varying numbers of training samples and fixed rollout budgets.
Results
CORE outperforms baseline methods across all tasks, achieving higher accuracy with fewer rollouts. Average accuracy improvement exceeds 50%.
Applications
CORE can be applied to tasks requiring rapid reasoning improvement, such as logic, planning, and problem-solving, reducing rollout demands and suitable for resource-limited scenarios.
Limitations & Outlook
CORE still requires a substantial number of initial samples to estimate baseline accuracy. Without similar successful cases, it may struggle to generate useful insights.
Plain Language Accessible to non-experts
Imagine a teacher helping students solve problems. The teacher encourages students to review past successes and failures to identify effective strategies. CORE acts like this teacher, helping the model learn and improve by contrasting successful and unsuccessful reasoning traces.
ELI14 Explained like you're 14
CORE is like a game guide, helping you quickly find ways to beat the game. By comparing your past successes and failures, it generates useful tips for solving similar problems next time. Isn't that cool?
Glossary
Contrastive Reflection
A learning method that generates insights by contrasting successful and unsuccessful cases.
CORE uses contrastive reflection to generate natural language insights.
Insight Memory
An external memory store for storing model-generated natural language insights.
CORE uses insight memory to retrieve and apply insights.
Rollout Memory
An external memory store for storing past model rollouts.
CORE uses rollout memory to contrast successful and unsuccessful cases.
Utility Estimates
Metrics used to evaluate the effectiveness of insights in problem-solving.
CORE retrieves insights based on utility estimates.
Non-parametric Learning
A method that improves models without updating weights.
CORE is a non-parametric learning algorithm.
Open Questions Unanswered questions from this research
- 1 How to generate effective insights without similar successful cases?
- 2 How does CORE perform on large-scale tasks?
Applications
Immediate Applications
Education
CORE can be used in education to help students quickly improve problem-solving skills.
Long-term Vision
Smart Assistants
CORE can be used to develop smarter assistants that help users quickly find solutions in complex tasks.
Abstract
Language models can use verifiable rewards to improve at a wide variety of reasoning tasks. However, both parametric (e.g. RLVR) and non-parametric (e.g. prompt optimization) approaches to doing so typically require hundreds of training samples and thousands of model rollouts, making them expensive in the best case and intractable in the worst. To address this challenge, we introduce Contrastive Reflection (CORE), a non-parametric learning algorithm that compares past reasoning traces to generate insights: short natural-language descriptions of reasoning strategies and constraints that capture differences between successful and unsuccessful problem attempts. Across four reasoning tasks, we demonstrate that CORE enables more rapid improvement than both parametric (GRPO) and non-parametric (GEPA, episodic RAG, and MemRL) methods, while using fewer rollouts. Under fixed rollout budgets with as few as five training samples, CORE achieves the strongest performance in most task-data regimes. Finally, we highlight how CORE is substantially more context-efficient than non-parametric baselines, requiring fewer prompt tokens while storing learned knowledge as compact, interpretable natural-language insights. Our results therefore suggest that distilling contrasts between successful and unsuccessful reasoning traces into abstract and useful insights can provide a more efficient and interpretable route to model self-improvement than weight updates, prompt optimization, or direct reuse of stored reasoning traces.