Kimi k1.5: Scaling Reinforcement Learning with LLMs
Kimi k1.5 scales LLMs with RL, achieving 77.5 on AIME and other top scores.
Key Findings
Methodology
Kimi k1.5 employs reinforcement learning with long context scaling, using online mirror descent for policy optimization. Trained on multimodal data, it excels in text and vision tasks. Key is avoiding complex techniques like Monte Carlo tree search, instead leveraging long context and policy optimization for performance gains.
Key Results
- Achieved 77.5 on AIME, 96.2 on MATH 500, 94th percentile on Codeforces, 74.9 on MathVista, matching OpenAI's o1.
- Short-CoT models scored 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench, outperforming GPT-4o and Claude Sonnet 3.5.
- Long-CoT techniques significantly enhance short-CoT model performance, boosting reasoning capabilities.
Significance
The study demonstrates the potential of scaling LLMs with reinforcement learning, overcoming traditional dataset limitations. Its successful application in multimodal tasks marks a new advancement in AI for complex reasoning tasks, offering new research directions for academia and industry.
Technical Contribution
Kimi k1.5 introduces long context scaling and improved policy optimization methods, simplifying the RL framework. Trained on multimodal data, the model achieves leading performance across benchmarks, offering new engineering possibilities.
Novelty
First to apply long context scaling in RL for LLMs, simplifying the need for complex techniques and providing an effective policy optimization method, showing significant innovation compared to existing methods.
Limitations
- The model may underperform on specific tasks, especially those requiring fine-grained reasoning.
- Handling long contexts demands high computational resources, potentially limiting practical applications.
Future Work
Future research could explore more efficient context handling methods, further optimize policy algorithms, and expand to more modalities and tasks.
AI Executive Summary
Kimi k1.5 scales large language models using reinforcement learning, addressing the limitations of traditional methods constrained by static datasets. By employing long context scaling and training on multimodal data, the model achieves leading scores across multiple benchmarks, demonstrating its potential in complex reasoning tasks.
The approach uses online mirror descent for policy optimization, combined with long context scaling, simplifying the need for traditional complex techniques like Monte Carlo tree search and value functions. Through multimodal training, the model excels in both text and vision tasks.
Despite significant progress, Kimi k1.5 faces challenges such as high computational demands and suboptimal performance on specific tasks. Future research will focus on optimizing context handling and policy algorithms to expand its application scope.
Deep Analysis
Background
In recent years, large language models (LLMs) have rapidly developed, but their training typically relies on large high-quality datasets. Traditional methods like next token prediction are effective for scaling compute but limited by dataset availability. Reinforcement learning offers a new scaling axis by guiding models to explore with rewards, breaking dataset limitations.
Core Problem
Traditional LLM training is limited by static datasets, unable to fully leverage dynamic environment information. How to expand LLM capabilities with reinforcement learning without relying on complex techniques is a pressing issue.
Innovation
Kimi k1.5's core innovations include long context scaling and improved policy optimization methods. By extending the context window, the model can reason and plan more effectively, while the online mirror descent algorithm provides robust policy optimization. Multimodal data training further enhances the model's adaptability.
Methodology
- �� Long Context Scaling: Extends context window to 128k, improving reasoning efficiency.
- �� Improved Policy Optimization: Uses online mirror descent algorithm, combined with effective sampling strategies and length penalties.
- �� Multimodal Data Training: Combines text and vision data, enhancing comprehensive reasoning capabilities.
Experiments
Experiments use multiple benchmark datasets, including AIME, MATH 500, and Codeforces. The model achieves leading scores on these datasets, validating the method's effectiveness. Key hyperparameters include context window size and policy optimization step size.
Results
Kimi k1.5 achieves significant results across multiple benchmarks, such as 77.5 on AIME, 96.2 on MATH 500, and 94th percentile on Codeforces, demonstrating its potential in complex reasoning tasks.
Applications
The method can be applied to tasks requiring complex reasoning, such as math competition problem solving and code generation. Its multimodal capability allows it to excel in both text and vision tasks.
Limitations & Outlook
Despite significant progress, Kimi k1.5 may underperform on specific tasks and requires high computational resources for long context handling. Future research will focus on optimizing these aspects.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking. Kimi k1.5 is like a smart kitchen assistant that not only remembers recipes but also adjusts them according to your taste. Through reinforcement learning, it learns to make the best choices among different ingredients and tools, just like finding the best solutions in different tasks. The long context is like its memory, helping it remember more steps and details, while policy optimization is how it continually improves its cooking skills.
ELI14 Explained like you're 14
Imagine you're playing a super complex game, and Kimi k1.5 is your game assistant. It not only remembers the game rules but also gives suggestions based on your play style. Through reinforcement learning, it learns how to find the best strategies in different levels, just like finding the best solutions in different tasks. The long context is like its memory, helping it remember more game details, while policy optimization is how it continually improves its gaming skills.
Glossary
Reinforcement Learning
A machine learning method that guides models to learn through reward mechanisms.
Used to scale training data for large language models.
Long Context
A method to extend the model's context window to improve reasoning ability.
Used to enhance Kimi k1.5's reasoning efficiency.
Policy Optimization
An algorithm to improve the decision-making ability of models.
Implemented through online mirror descent.
Multimodal
A training method combining multiple data types.
Used to enhance model performance in text and vision tasks.
Online Mirror Descent
An algorithm used for policy optimization.
Used to improve Kimi k1.5's policy optimization.
Open Questions Unanswered questions from this research
- 1 How to further extend the context window without increasing computational resources?
- 2 How to improve model performance on specific tasks?
Applications
Immediate Applications
Math Competition Problem Solving
Enhances model performance in math problem solving through reinforcement learning, applicable in competitions and education.
Code Generation
Utilizes multimodal capabilities to improve model performance in code generation tasks, applicable in software development.
Long-term Vision
General Artificial Intelligence
Advances towards general AI by continually optimizing policy and context handling.
Abstract
Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).