Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM
Steel-LLM uses Soft MoE and enhanced FFN, excelling in CEVAL benchmarks.
Key Findings
Methodology
Steel-LLM utilizes Soft Mixture of Experts and enhanced Feed-Forward Network, focusing on Chinese data training. By optimizing the TinyLlama framework, it enhances training efficiency and model performance. Techniques like FlashAttention and mixed precision training ensure effective training under limited resources.
Key Results
- Steel-LLM achieved 41.90% accuracy on CEVAL benchmarks, surpassing early models from larger institutions.
- On CMMLU tests, it outperformed models like Gemma-2b-it and Llama2-7B.
- Various fine-tuning strategies optimized the model's multilingual capabilities and reasoning abilities.
Significance
This research provides practical guidance for resource-limited teams to develop high-quality LLMs, especially in the Chinese domain. By offering a fully open-source training pipeline and dataset, it promotes transparency and collaboration within the community.
Technical Contribution
Steel-LLM achieves performance comparable to larger models under limited resources, employing innovative Soft MoE structure and enhanced FFN, offering new engineering possibilities.
Novelty
First to apply Soft MoE structure in the Chinese domain, optimizing training efficiency and performance, filling gaps in open-source LLMs for Chinese data processing.
Limitations
- The model still faces limitations in handling multilingual data, especially with a low proportion of English data.
- Uneven data distribution during training may affect model performance.
Future Work
Future directions include optimizing data distribution, enhancing multilingual capabilities, and exploring more resource-efficient training methods.
AI Executive Summary
The Steel-LLM project aims to address the shortcomings of existing open-source LLMs in transparency and resource efficiency. By employing Soft Mixture of Experts and enhanced Feed-Forward Network, Steel-LLM achieves a high-quality Chinese language model under limited computational resources. Experimental results show that Steel-LLM performs excellently on CEVAL and CMMLU benchmarks, surpassing early models from larger institutions. This research provides practical guidance for resource-limited teams to develop high-quality LLMs, especially in the Chinese domain. While the model still faces limitations in handling multilingual data, its fully open-source training pipeline and dataset promote transparency and collaboration within the community. Future directions include optimizing data distribution, enhancing multilingual capabilities, and exploring more resource-efficient training methods.
Deep Analysis
Background
Recent years have seen significant advancements in open-source large language models in natural language processing, with applications ranging from conversational agents to code generation. However, many prominent models still face challenges in transparency and resource efficiency, limiting the development capabilities of smaller research teams.
Core Problem
Existing open-source LLMs often lack transparent training data and intermediate checkpoints, hindering model reproducibility and further research by the community. This limited openness makes it difficult for smaller teams to develop high-quality models in non-English languages, such as Chinese.
Innovation
Steel-LLM employs Soft Mixture of Experts and enhanced Feed-Forward Network, optimizing the TinyLlama framework to improve training efficiency. By offering a fully open-source training pipeline and dataset, it provides practical guidance for resource-limited teams to develop high-quality LLMs.
Methodology
- �� Use Soft Mixture of Experts to optimize the FFN layer, enhancing model performance.
- �� Employ FlashAttention and mixed precision training techniques to improve training efficiency.
- �� Optimize the TinyLlama framework to support data appending and training progress restoration.
Experiments
Experimental design includes pretraining using Chinese datasets like SkyPile-150B and Wanjuan1.0, and fine-tuning using datasets like Infinity-Instruct and Wanjuan-cn. AdamW optimizer and cosine annealing learning rate schedule are employed.
Results
Steel-LLM performs excellently on CEVAL and CMMLU benchmarks, surpassing early models from larger institutions. Fine-tuning strategies optimized the model's multilingual capabilities and reasoning abilities.
Applications
Steel-LLM can be used for Chinese natural language processing tasks, suitable for resource-limited research teams. Its open-source nature promotes collaboration and transparency within the community.
Limitations & Outlook
The model still faces limitations in handling multilingual data, especially with a low proportion of English data. Uneven data distribution during training may affect model performance.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking, Steel-LLM is like a smart chef. It has a special spice rack (Soft MoE) that can pick the best spices for each dish to enhance its flavor. This chef also adjusts cooking methods (enhanced FFN) based on different recipes to ensure every dish is made perfectly with limited kitchen equipment. Although it sometimes makes dishes that aren't as flavorful due to limited ingredients, it always maximizes its potential under limited conditions.
ELI14 Explained like you're 14
Imagine you're playing a super cool game, Steel-LLM is your game character. It has a special skill tree (Soft MoE) that can choose the best skills for different tasks to complete challenges. This character also adjusts attack methods (enhanced FFN) based on different levels to ensure it defeats enemies with limited resources. Although it sometimes doesn't perform well due to a lack of items, it always maximizes its potential under limited conditions.
Glossary
Soft Mixture of Experts
A model structure that allows multiple expert networks to participate in decision-making, enhancing model performance.
Used to optimize the FFN layer in Steel-LLM.
FlashAttention
An efficient attention mechanism that improves training and inference efficiency.
Used in Steel-LLM's self-attention module.
Feed-Forward Network
A neural network structure that processes input data and generates output.
Steel-LLM employs an enhanced FFN.
AdamW Optimizer
An optimization algorithm combining Adam and weight decay techniques.
Used in Steel-LLM's training process.
Cosine Annealing Learning Rate Schedule
A learning rate adjustment strategy that gradually reduces the learning rate to stabilize training.
Used during Steel-LLM's training phase.
Open Questions Unanswered questions from this research
- 1 How to further optimize multilingual data processing to enhance the model's multilingual capabilities?
- 2 How to improve training efficiency under limited resources?
Applications
Immediate Applications
Chinese Natural Language Processing
Steel-LLM can be used for Chinese text generation and understanding tasks, suitable for resource-limited teams.
Open Source Research Collaboration
By offering a fully open-source training pipeline and dataset, it promotes transparency and collaboration within the community.
Long-term Vision
Multilingual Model Development
Explore more resource-efficient training methods to enhance the model's multilingual capabilities.
Abstract
Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.