Textbooks Are All You Need II: phi-1.5 technical report
phi-1.5 model achieves comparable common sense reasoning with 1.3B parameters as models 5x larger.
Key Findings
Methodology
The phi-1.5 model uses a Transformer architecture with 24 layers and 32 heads, each with a dimension of 64. It employs rotary embedding and flash-attention for training acceleration. The training data consists of 7B tokens from phi-1 and 20B newly created synthetic 'textbook-like' data, focusing on common sense reasoning and world knowledge.
Key Results
- phi-1.5 performs comparably to Llama2-7B and Vicuna-13B on common sense reasoning benchmarks like WinoGrande, especially excelling in multi-step reasoning tasks.
- In GSM8K math and HumanEval coding tasks, the phi-1.5-web model surpasses Llama 65B, demonstrating efficient knowledge storage and access.
- phi-1.5 using synthetic data shows better performance in reducing toxic content generation compared to models trained solely on web data.
Significance
The research on phi-1.5 demonstrates the potential of small-scale models in common sense reasoning and complex tasks, challenging the dominance of large-scale models. By using synthetic data, phi-1.5 advances in reducing toxic and biased content generation, offering new insights for responsible AI development.
Technical Contribution
Phi-1.5 showcases the potential of small models in complex tasks through synthetic data training, proposing new data generation and selection strategies, reducing reliance on large-scale web data, and providing new engineering possibilities.
Novelty
Phi-1.5 is the first to achieve comparable common sense reasoning capabilities in a small-scale model, innovatively using synthetic data to replace traditional web data, reducing the risk of generating toxic content.
Limitations
- Phi-1.5 may still underperform in certain complex tasks, especially those requiring extensive background knowledge.
- The model may exhibit hallucinations when dealing with unconventional inputs.
- Lacks instruction fine-tuning, potentially limiting instruction-following performance.
Future Work
Future work can explore optimization strategies for synthetic data generation, further reducing toxic content generation, and delve into instruction fine-tuning and RLHF applications.
AI Executive Summary
In recent years, large language models have made significant advances in natural language processing, but their massive scale brings high computational costs and energy consumption. The phi-1.5 model, trained with synthetic data, demonstrates the potential of small-scale models in common sense reasoning and complex tasks.
Phi-1.5 employs a Transformer architecture, combining rotary embedding and flash-attention techniques, and is trained with synthetic 'textbook-like' data, focusing on common sense reasoning and world knowledge. Experimental results show that phi-1.5 performs excellently across multiple benchmarks, particularly excelling in multi-step reasoning tasks.
Despite phi-1.5's progress in reducing toxic content generation, it still has limitations, such as potential hallucinations when handling unconventional inputs. Future research can further optimize synthetic data generation strategies and explore instruction fine-tuning and RLHF applications.
Deep Analysis
Background
In recent years, Transformer-based language models have achieved significant progress in natural language processing. Representative works include large-scale models like GPT-3 and PaLM, which perform well across multiple tasks but bring high computational costs and energy consumption. Phi-1.5 aims to explore the potential of small-scale models in common sense reasoning tasks.
Core Problem
The high computational costs and energy consumption of large language models limit their widespread application. How to achieve comparable capabilities in small-scale models, especially in common sense reasoning and complex tasks, is an important research question.
Innovation
The core innovation of phi-1.5 lies in using synthetic 'textbook-like' data for training, reducing reliance on large-scale web data. This approach advances in reducing toxic content generation while performing excellently in common sense reasoning tasks.
Methodology
- �� Uses Transformer architecture with 24 layers, 32 heads, each with dimension 64.
- �� Employs rotary embedding and flash-attention techniques for training acceleration.
- �� Training data includes 7B tokens from phi-1 and 20B newly created synthetic 'textbook-like' data.
- �� Focuses on common sense reasoning and world knowledge.
Experiments
The experimental design includes multiple benchmarks for common sense reasoning and complex tasks, such as WinoGrande, ARC-Easy, and GSM8K. Zero-shot evaluation is conducted using LM-Eval Harness, comparing phi-1.5's performance with other models.
Results
Phi-1.5 performs comparably to Llama2-7B and Vicuna-13B on common sense reasoning benchmarks, especially excelling in multi-step reasoning tasks. In GSM8K math and HumanEval coding tasks, the phi-1.5-web model surpasses Llama 65B.
Applications
Phi-1.5 can be applied in scenarios requiring common sense reasoning and complex task handling, such as intelligent assistants and educational tools. Its small scale and high efficiency make it advantageous in resource-limited environments.
Limitations & Outlook
Phi-1.5 may still underperform in certain complex tasks, especially those requiring extensive background knowledge. The model may exhibit hallucinations when dealing with unconventional inputs, and future research should further optimize synthetic data generation strategies.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking. Large models are like a big feast requiring many ingredients and time, while phi-1.5 is like a simple yet delicious home-cooked meal. By using high-quality synthetic ingredients, phi-1.5 can make a tasty dish without needing a lot of ingredients. It's like making a delicious meal with fewer ingredients instead of relying on a lot. Phi-1.5 excels in common sense reasoning and complex tasks, reducing toxic content generation.
ELI14 Explained like you're 14
Imagine you're playing a game where you usually need a super powerful character to defeat the big boss, but phi-1.5 is like a small character that can use smart strategies to beat the big boss. It uses clever synthetic data instead of relying on lots of web data to complete tasks. It's like winning a game with smart strategies instead of brute force. Phi-1.5 excels in common sense reasoning and complex tasks, just like defeating the big boss with a small character.
Glossary
Transformer
A deep learning model architecture used for natural language processing, adept at handling sequential data.
Phi-1.5 uses the Transformer architecture to achieve efficient common sense reasoning.
Common Sense Reasoning
The ability of AI to understand and infer everyday common-sense knowledge.
Phi-1.5 excels in common sense reasoning tasks, approaching the level of large-scale models.
Synthetic Data
Artificially generated data used for training AI models.
Phi-1.5 is trained with synthetic 'textbook-like' data, reducing toxic content generation.
Flash-Attention
A technique to accelerate Transformer model training by optimizing memory access.
Phi-1.5 employs flash-attention to speed up the training process.
Zero-shot Learning
The ability of AI to reason without having seen specific task data.
Phi-1.5 is evaluated zero-shot on multiple benchmarks, performing excellently.
Open Questions Unanswered questions from this research
- 1 How to further optimize synthetic data generation strategies to reduce toxic content generation?
- 2 In instruction fine-tuning and RLHF, how can phi-1.5 further improve instruction-following capabilities?
Applications
Immediate Applications
Intelligent Assistants
Phi-1.5 can be used to develop smarter assistants, providing more accurate common sense reasoning and task execution capabilities.
Long-term Vision
Educational Tools
Phi-1.5 can be used to develop educational tools, helping students improve learning outcomes in common sense reasoning and complex tasks.
Abstract
We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce coherent English -- and the follow-up work on \textbf{phi-1}, a 1.3 billion parameter model with Python coding performance close to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to generate ``textbook quality" data as a way to enhance the learning process compared to traditional web data. We follow the ``Textbooks Are All You Need" approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named \textbf{phi-1.5}, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. More generally, \textbf{phi-1.5} exhibits many of the traits of much larger LLMs, both good -- such as the ability to ``think step by step" or perform some rudimentary in-context learning -- and bad, including hallucinations and the potential for toxic and biased generations -- encouragingly though, we are seeing improvement on that front thanks to the absence of web data. We open-source \textbf{phi-1.5} to promote further research on these urgent topics.