TinyLlama: An Open-Source Small Language Model
TinyLlama trains a 1.1B Llama-2-style model on up to 3T tokens, outperforming similarly sized open baselines.
Key Findings
Methodology
TinyLlama uses a Llama 2 decoder-only Transformer and tokenizer: 22 layers, 2,048 hidden units, 2,048-token context, 32 query heads, 4 key-value heads, and a 32,000-token vocabulary. Its corpus combines SlimPajama and StarCoder at roughly 7:3, totaling about 950B tokens; the original model processes approximately 3T tokens over three epochs. RoPE, RMSNorm, SwiGLU, and Grouped-Query Attention are paired with FlashAttention-2, FSDP, xFormers, and Lit-GPT for efficient training.
Key Results
- On zero-shot commonsense tasks, TinyLlama v1.1 reaches an average of 53.63, exceeding OPT-1.3B at 51.44 and Pythia-1.4B at 51.33. Its HellaSwag score is 61.47, indicating that extensive data can substantially strengthen a small model.
- The Math&Code variant averages 21.18 on InstructEval, compared with 17.72 for Pythia-1.4B. It obtains 15.24 on HumanEval and 18.54 on DROP, showing clear gains from domain-focused continual pretraining.
- Engineering optimizations achieve about 24,000 tokens/s per A100-40G. Training 300B tokens requires 3,456 GPU hours, versus 4,830 for Pythia-1.0B and 7,920 for MPT-1.3B.
Significance
The paper shows that a 1.1B model need not rely exclusively on parameter scaling. High-quality data, prolonged training, and optimized systems allow a small model to match or surpass comparable open models across commonsense, coding, mathematical, and Chinese tasks. This lowers the barrier to reproducible pretraining research and supports mobile or edge deployment. More broadly, the work shifts attention from parameter count alone toward jointly optimizing model size, data volume, training duration, and inference cost.
Technical Contribution
The work provides a reusable 1.1B pretraining stack. Grouped-Query Attention reduces KV-cache bandwidth; FlashAttention-2 improves memory-efficient attention; FSDP enables distributed parameter sharding; and xFormers supplies fused SwiGLU kernels. TinyLlama v1.1 adds a three-stage schedule—basic pretraining, domain-specific continual pretraining, and cooldown. Instead of only lowering the learning rate at the end, it increases batch size from about 1.8M to 7.2M tokens, offering a practical convergence strategy.
Novelty
The novelty is primarily systemic rather than a new attention equation. The authors describe this as the first attempt to train a roughly 1B-parameter model on up to 3T tokens. By combining Llama 2 components, carefully composed open data, efficient kernels, and released checkpoints, TinyLlama turns the “small model plus very large corpus” hypothesis into a reproducible experimental platform, unlike conventional OPT or Pythia-scale comparisons.
Limitations
- Evaluation emphasizes zero-shot commonsense, few-shot InstructEval, and Chinese multiple-choice tests. It does not comprehensively measure deployment robustness, factuality, safety, contamination, or long-context behavior.
- Although more efficient than baselines, reproduction still requires substantial hardware: the original uses 16 A100-40G GPUs, while v1.1 uses 64. The paper also provides limited independent ablations of its training stages.
Future Work
Future work should study data contamination, longer contexts, instruction tuning, safety alignment, and controlled ablations of repetition and domain mixtures. Quantized deployment should be evaluated on phones and edge devices using latency, energy, accuracy, and reliability metrics. More complete scaling laws for small models could connect training cost to real-world utility.
AI Executive Summary
Large language models have achieved strong results by expanding parameter counts, but their training and inference costs restrict access. TinyLlama explores a different regime: a 1.1B-parameter model trained on roughly 950B unique tokens and, in the original run, about 3T cumulative tokens. The study asks whether data and training efficiency can compensate for limited capacity.
The model follows Llama 2’s decoder-only Transformer and tokenizer, using RoPE, RMSNorm, SwiGLU, and Grouped-Query Attention. Its training stack combines FlashAttention-2, FSDP, xFormers, and Lit-GPT. TinyLlama reaches about 24,000 tokens/s per A100-40G. Version 1.1 introduces basic pretraining, domain-specific continual pretraining, and a batch-size-based cooldown, producing general, Math&Code, and Chinese variants.
The results are competitive: TinyLlama v1.1 averages 53.63 across seven zero-shot commonsense tasks, above OPT-1.3B’s 51.44 and Pythia-1.4B’s 51.33. Math&Code averages 21.18 on InstructEval, with 15.24 on HumanEval, while the Chinese variant averages 58.37 on Chinese benchmarks. The contribution is therefore an open, efficient research platform rather than a new fundamental architecture. Its limitations remain important: evaluation is narrow, full reproduction is hardware-intensive, and safety, contamination, long-context reliability, and real-device performance are insufficiently established.
Deep Analysis
Background
NLP has evolved from GPT-style scaling to systems such as PaLM and Llama that combine larger models with larger corpora. Kaplan et al. formalized scaling behavior, while Hoffmann et al. proposed compute-optimal allocation. Llama research also showed that smaller models trained on more data can be attractive under inference constraints. Open models such as OPT and Pythia improve accessibility, but comparable small models still face quality, memory, and training-throughput limitations.
Core Problem
The central question is whether a roughly 1B model can become competitive when trained on far more data than standard scaling prescriptions suggest. The challenge involves constructing clean mixed-domain data, controlling duplication and contamination, stabilizing long training runs, overcoming memory and communication bottlenecks, and preserving balanced performance across English, code, mathematics, and Chinese.
Innovation
- �� Reuses the Llama 2 architecture and tokenizer for compatibility and accessibility.
- �� Mixes SlimPajama and StarCoder at approximately 7:3 and processes up to 3T tokens.
- �� Integrates FlashAttention-2, FSDP, fused normalization, fused cross-entropy, and xFormers kernels.
- �� Version 1.1 uses 1.5T-token basic training, 350B-token domain training, and a 150B-token cooldown, yielding general, Math&Code, and Chinese variants.
Methodology
- �� Input: cleaned SlimPajama text plus StarCoder code and text-code data, tokenized with the Llama tokenizer.
- �� Model: 22-layer decoder-only Transformer; hidden size 2,048; intermediate size 5,632; context length 2,048; 32 query heads and 4 KV heads; RoPE positions.
- �� Optimization: AdamW with β1=0.9 and β2=0.95; cosine learning rate from 4e-4 to 4e-5; 2,000 warmup steps; weight decay 0.1; gradient clipping 1.0.
- �� Systems: the original run uses 16 A100-40G GPUs; FSDP and fused kernels reduce memory and communication costs.
- �� v1.1: batch size rises from about 1.8M to 7.2M tokens during cooldown.
Experiments
Baselines are OPT-1.3B, Pythia-1.0B, and Pythia-1.4B. Commonsense evaluation uses Language Model Evaluation Harness on HellaSwag, OpenBookQA, WinoGrande, ARC-Easy, ARC-Challenge, BoolQ, and PIQA in zero-shot mode. InstructEval uses five-shot MMLU, three-shot BBH and DROP, and zero-shot HumanEval. Chinese evaluation covers xwinograd, xstorycloze, XNLI, and XCOPA.
Results
TinyLlama v1.1 averages 53.63 on commonsense tasks, exceeding OPT-1.3B’s 51.44. Math&Code reaches 15.24 on HumanEval, 18.54 on DROP, and 21.18 overall on InstructEval. The Chinese variant averages 58.37 on Chinese tasks, versus 47.46 for general v1.1. For 300B training tokens, TinyLlama requires 3,456 GPU hours, compared with 4,830 for Pythia-1.0B and 7,920 for MPT-1.3B.
Applications
TinyLlama can support local assistants, lightweight code completion, educational experiments, domain prototypes, and edge-device research. Researchers can use released checkpoints to study data mixtures, continual pretraining, quantization, and alignment. Industrial deployment would still require instruction tuning, safety filtering, domain validation, and careful latency and reliability testing.
Limitations & Outlook
The 1.1B capacity constrains difficult reasoning, factual coverage, and generation reliability; it should not be equated with frontier models. The paper does not systematically report bias, hallucination, jailbreak resistance, contamination, or long-context results. Chinese gains may partly reflect Chinese text embedded in Python and Jupyter data. Full training remains expensive, and the three-stage schedule lacks extensive isolated ablations. Future work should evaluate cost-quality tradeoffs on real devices.
Plain Language Accessible to non-experts
Imagine a small library trying to compete with a giant national library. The giant has far more shelves, but it is expensive to build and operate. TinyLlama has only 1.1 billion “storage slots,” so its creators compensate by giving it a carefully cleaned collection of books, websites, programs, and mathematics. SlimPajama supplies broad reading; StarCoder supplies programming material. The model reads the collection repeatedly, much like a small study group reviewing the same high-quality notes.
The engineers also redesign the library’s workflow. FlashAttention-2 reduces unnecessary movement of information, Grouped-Query Attention lets several questions share the same reference material, and FSDP divides the work across machines. The result is a small system that scores 53.63 on average across commonsense tests, ahead of larger models such as OPT-1.3B and Pythia-1.4B.
Version 1.1 resembles a school curriculum: general lessons first, then specialized courses in code, mathematics, or Chinese, followed by a final consolidation period. The lesson is simple: performance depends not only on size, but also on the quality of materials, amount of practice, and organization of learning.
ELI14 Explained like you're 14
Think of TinyLlama as a small game bot with 1.1 billion memory slots. It is much smaller than the giant AI bots you hear about, but researchers make it study a huge amount of text, code, and math—then review the material several times. SlimPajama is like a giant library, while StarCoder is like a programming club’s notebook.
First, the bot learns general knowledge. Then special versions train harder on coding and math or on Chinese. The team also uses clever speed tricks: FlashAttention-2 is like taking fewer trips across a classroom, and FSDP is like asking many teammates to share the homework. Less wasted movement means faster training.
The scores are surprisingly strong: TinyLlama v1.1 averages 53.63 on commonsense tests, above OPT-1.3B’s 51.44. The Math&Code version gets 15.24 on HumanEval. Pretty cool, right? But it is not magic. A small bot can still forget facts, misunderstand tricky questions, or produce unsafe answers, so people must test and supervise it before using it seriously.
Glossary
Decoder-only Transformer
A neural architecture that predicts the next token from previous tokens. Technically, it uses causal self-attention rather than a separate encoder-decoder pathway.
It is TinyLlama’s core model architecture.
RoPE
Rotary Position Embedding injects token position by rotating query and key representations. This lets attention account for sequence order.
It supplies positional information in every Transformer layer.
Grouped-Query Attention
Multiple query heads share a smaller number of key-value heads. This reduces KV-cache memory and inference bandwidth with limited quality loss.
TinyLlama uses 32 query heads and 4 KV heads.
FlashAttention-2
An attention algorithm that uses tiling and optimized memory access to reduce data movement. It preserves exact attention computation while improving speed and memory efficiency.
It is a major training-throughput optimization.
Continual pretraining
Further pretraining an existing general model on domain-specific data. It adapts capabilities without building a separate architecture.
v1.1 uses it for Math&Code and Chinese specialization.
Open Questions Unanswered questions from this research
- 1 The long-term effects of repeated data exposure on memorization, overfitting, and contamination remain unclear. Controlled repetition studies and transparent deduplication analyses are needed.
- 2 The paper does not establish quantized latency, energy use, reliability, or safety on real phones and edge devices, so practical deployment value remains uncertain.
Applications
Immediate Applications
Local lightweight assistant
Developers can combine the released checkpoints with quantization and instruction tuning for low-latency summarization, classification, or simple question answering. They must still test privacy, domain accuracy, and hallucination rates.
Coding and education prototypes
The Math&Code model can seed code completion, exercise generation, and classroom experiments. Its modest size makes experimentation more accessible to students and small research teams with limited GPU budgets.
Long-term Vision
Edge-model ecosystem
With further quantization, distillation, context extension, and safety alignment, TinyLlama-like systems could run on phones and offline devices. Reliability, multilingual coverage, and adversarial robustness remain the main obstacles.
Abstract
We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama.