PaLM 2 Technical Report
PaLM 2 combines UL2-style objectives, compute-optimal scaling, and multilingual data to improve reasoning and translation with smaller, faster models.
Key Findings
Methodology
PaLM 2 uses a Transformer architecture trained with a tuned mixture of objectives inspired by UL2 rather than a single causal objective. Its corpus combines web documents, books, code, mathematics, dialogue, and parallel text spanning hundreds of languages, with deduplication, quality filtering, and sensitive-PII removal. Scaling experiments apply FLOPs≈6ND across budgets from 1×10^19 to 1×10^22 FLOPs. Quadratic fits to IsoFLOP curves estimate the parameter/data balance that minimizes validation loss.
Key Results
- In 1-shot English evaluation, PaLM 2-L averages 76.9%, versus 70.4% for PaLM. It reaches 86.1% on TriviaQA, 62.3% on RACE-H, and 67.1% on ANLI-R3, compared with PaLM scores of 81.4%, 52.1%, and 52.3%, respectively.
- PaLM 2 passes every evaluated advanced language exam: Chinese HSK 7–9 overall 82%, Japanese J-Test 94%, French TCF 82%, and Spanish DELE C2 overall 62%. The report also describes substantial gains in reasoning, translation, coding, and natural-language generation.
- At 1×10^22 FLOPs, approximately 9.5B parameters minimize training loss, close to the reported PaLM 2 optimum of 9.50B. Yet a 16.1B model slightly outperforms it on some downstream tasks, showing that loss is not a perfect task-performance proxy.
Significance
The report’s central significance is that capability need not come primarily from parameter expansion. Higher-quality, more diverse multilingual data, compute-optimal data/model ratios, and objective design allow smaller models to improve language understanding, reasoning, and translation simultaneously. PaLM 2-L is smaller than the largest PaLM model but uses more training compute and achieves stronger results, improving serving latency and cost. For industry, this supports broader deployment; for research, it reinforces that data quality, training objectives, and systems efficiency are as important as raw scale.
Technical Contribution
The technical contributions are threefold. First, large-scale experiments independently support proportional growth of data D and parameters N, contrasting with earlier parameter-heavy Kaplan scaling. Second, PaLM 2 integrates UL2-style objective mixtures with multilingual parallel data, code, and mathematics to improve transfer and reasoning. Third, a small fraction of training examples receives Perspective API toxicity-control labels, enabling inference-time toxicity conditioning without additional decoding overhead or measurable unrelated-task damage. Multilingual canary sequences provide a mechanism for estimating verbatim memorization across languages.
Novelty
PaLM 2 is not a radically new neural architecture; its novelty is a system-level integration of scaling science, data engineering, objective mixtures, long context, and safety controls. Compared with PaLM-like systems dominated by English data and a single language-modeling objective, it trains smaller models on richer multilingual and domain-specific mixtures and demonstrates joint gains in capability, efficiency, and controllability.
Limitations
- Many evaluations use few-shot prompts or simulated examinations. Writing is human-rated, speaking is omitted, and reported exam outcomes are not official certifications; benchmark gains therefore do not fully establish real-world communicative competence.
- The complete corpus, training configuration, and all model details are not disclosed. User-facing products add preprocessing and postprocessing and may change over time, so product behavior cannot be inferred exactly from the reported checkpoint results.
- Scaling optimizes validation loss rather than every downstream objective: the 9.5B loss-optimal model slightly trails 16.1B on some tasks. Deployment decisions must jointly consider capability, throughput, latency, and cost.
Future Work
Future work should expand evaluation for low-resource languages and culturally situated use, isolate the causal contributions of each objective and data source, and improve long-context, factuality, privacy, and bias testing. A unified scaling theory should include task performance, serving latency, throughput, safety, and cost. More granular multilingual safety conditioning and transparent reproducibility practices are also needed.
AI Executive Summary
Large language models have often improved by adding parameters and predominantly English training text, but that strategy is expensive and leaves multilingual competence, reasoning, and deployment efficiency unresolved. The PaLM 2 report presents a broader recipe: a Transformer trained with a tuned mixture of objectives, a higher-quality corpus spanning hundreds of languages and specialized domains, and compute-optimal scaling. Using FLOPs≈6ND and IsoFLOP analysis, the authors find that model parameters and training data should grow at roughly comparable rates.
PaLM 2 is evaluated in Small, Medium, and Large variants. Its corpus includes web documents, books, dialogue, code, mathematics, and English-aligned parallel text, after deduplication and filtering. PaLM 2-L achieves 76.9% average accuracy on 1-shot English tasks, compared with 70.4% for PaLM; it scores 86.1% on TriviaQA and 67.1% on ANLI-R3. Its multilingual results are equally notable: 82% on Chinese HSK 7–9 overall, 94% on Japanese J-Test, and 82% on French TCF.
The report also addresses deployment and safety. Toxicity-control tokens permit inference-time steering without extra inference overhead or clear damage to unrelated abilities, while multilingual canaries measure memorization and privacy risk. Important caveats remain: simulated exams omit speaking, benchmark scores do not guarantee factual reliability, and products may evolve beyond the reported checkpoint. Overall, PaLM 2 argues that careful data selection, objective design, and efficient scaling can outperform brute-force enlargement.
Deep Analysis
Background
Language modeling progressed from n-grams and LSTMs to Transformers. GPT-3, PaLM, and Chinchilla showed that scale improves language and reasoning, while Kaplan et al. emphasized parameter growth and Hoffmann et al. argued for more balanced data/model scaling. PaLM 2 extends this agenda to multilinguality, code, mathematics, long context, safety, and serving efficiency.
Core Problem
Earlier LLMs were often English-dominated and trained with one causal or masked objective. This limits low-resource language coverage, cross-lingual transfer, and complex reasoning while making inference expensive. The challenge is to improve capability, context handling, safety, memorization measurement, and deployment cost together rather than optimizing only parameter count.
Innovation
- ��A broader multilingual and domain mixture with English-aligned parallel documents; •a tuned UL2-inspired mixture of pre-training objectives; •large-scale IsoFLOP experiments to estimate compute-optimal parameter/data ratios; •substantially longer context; •toxicity-control tokens for inference-time steering and multilingual canaries for memorization analysis. The contribution is primarily a coordinated training-and-systems design rather than a new backbone.
Methodology
- ��Data: combine web, books, code, mathematics, dialogue, monolingual multilingual text, and parallel text; deduplicate, filter quality, and remove sensitive PII.
- ��Pre-training: use a Transformer with mixed causal/denoising-style objectives, then apply generic instruction fine-tuning.
- ��Scaling: train 400M–15B parameter studies under 1×10^19–1×10^22 FLOPs; estimate tokens through FLOPs≈6ND and fit quadratic minima to IsoFLOP bands.
- ��Evaluation: compare PaLM 2-S/M/L with PaLM using few-shot QA, reasoning, coding, translation, generation, human exams, safety, and memorization tests.
Experiments
The evaluation covers TriviaQA, Natural Questions, HellaSwag, RACE, PIQA, ARC, SuperGLUE, ANLI, TyDi QA, BIG-Bench, HumanEval, translation benchmarks, and professional exams including HSK, J-Test, PLIDA, TCF, and DELE. Results generally use few-shot prompting and average the five latest PaLM 2-L checkpoints. Writing exams receive ratings from three professional native speakers; listening is represented by transcripts.
Results
PaLM 2-L averages 76.9% on English 1-shot tasks versus PaLM’s 70.4%. ANLI-R1/R2/R3 scores are 73.1%, 63.4%, and 67.1%, compared with 52.6%, 48.7%, and 52.3% for PaLM. In no-context TyDi QA, English reaches 43.7 F1, Finnish 45.5, and Indonesian 46.4. The loss-optimal point is near 9.5B parameters, but downstream rankings can favor larger models.
Applications
The model supports multilingual question answering, translation, education, code repair, mathematical explanation, long dialogue, and summarization. Smaller high-quality variants can reduce latency and serving costs. Toxicity controls may assist safer generation, but production systems still require moderation, access controls, privacy protection, testing, and human oversight.
Limitations & Outlook
Performance depends on data coverage, so low-resource languages and cultural viewpoints may remain uneven. Simulated language examinations omit speaking and are not official certification; benchmark scores do not establish factuality or robust safety. The report describes the May 2023 version, while products can change. Future work should improve transparency, causal ablations, long-context evaluation, privacy testing, and culturally grounded safety measurement.
Plain Language Accessible to non-experts
Imagine PaLM 2 as a huge school. An older school might collect mostly English textbooks and become smarter by adding more classrooms. PaLM 2 changes both the books and the lessons: it reads many languages, mathematics, programming, novels, web pages, and conversations. Some books come in matched pairs, such as English beside another language, so students learn how ideas correspond rather than merely memorizing isolated phrases.
The school also has a limited budget. Researchers compare different class sizes and amounts of reading material. They find that the best bargain is usually to grow the class and the library together. A smaller, well-trained class can answer questions quickly and perform strongly, instead of building the biggest possible school at enormous cost.
The results are impressive: PaLM 2-L scores 76.9% across many English tasks, compared with 70.4% for PaLM, and reaches 86.1% on TriviaQA. It passes advanced exams in Chinese, Japanese, French, and Spanish. It also has a safety switch that can reduce toxic wording. Still, a test score is not perfection: it may misunderstand facts, reflect bias, or remember training text. Real users therefore need checking, privacy safeguards, and supervision.
ELI14 Explained like you're 14
Think of PaLM 2 as a gaming teammate trained in a giant practice arena. Instead of reading only English walkthroughs, it studies Chinese, Japanese, Arabic, code, math, stories, and chats. Some lessons show the same idea in two languages, which helps it translate and explain things instead of just guessing the next word.
The researchers also ask: should we make the teammate enormous? Not always! If you spend all your game coins on character size but give it too little practice, the result may be disappointing. Their experiments suggest that model size and training data should grow together. That makes a smaller teammate faster and cheaper to run.
The scores are strong: PaLM 2-L averages 76.9% on many English tasks, while PaLM gets 70.4%. It scores 86.1% on TriviaQA and 67.1% on ANLI-R3, a difficult reasoning test. It also passes advanced Chinese, Japanese, French, and Spanish exams. Pretty cool, right?
But it is not an all-knowing robot. The language exams did not test speaking, and real products can change. It may still make mistakes, produce harmful text, or repeat remembered material. So treat it like a very fast, well-read teammate—not an automatic truth machine. Check important answers, protect private information, and keep safety rules switched on!
Glossary
Transformer
A neural architecture that uses attention to connect relevant parts of a sequence. It enables parallel processing and is PaLM 2’s backbone.
Used for pre-training and text generation.
Mixture of Objectives
Training with several prediction or denoising tasks instead of one objective. Different tasks encourage complementary language abilities.
PaLM 2 uses a tuned UL2-inspired mixture.
Compute-optimal scaling
Choosing model size and training-token count jointly under a fixed compute budget. The report uses FLOPs≈6ND as a training-cost heuristic.
Used to estimate efficient parameter/data ratios.
IsoFLOP curve
A comparison of validation loss across model sizes while total compute remains fixed. Its minimum estimates the best size for that budget.
Applied across 1×10^19–1×10^22 FLOPs.
TyDi QA
A multilingual question-answering benchmark. Gold Passage supplies evidence, whereas no-context gives only the question.
Measures cross-lingual reading and closed-book QA.
BIG-Bench
A broad collection of difficult language-model tasks. It is especially useful for studying reasoning and generalization.
Used to demonstrate reasoning gains over PaLM.
Open Questions Unanswered questions from this research
- 1 Real-world performance in low-resource languages, dialects, and culturally specific settings remains uncertain. Existing benchmarks need longer-term evaluation led by local experts.
- 2 The separate contributions of objective mixtures, data quality, parallel text, and domain data are not fully isolated. Reproducible ablations are needed.
- 3 Validation loss does not perfectly predict downstream rankings. Future scaling laws should jointly model capability, latency, throughput, safety, and cost.
Applications
Immediate Applications
Multilingual education and translation
Education platforms can generate explanations, exercises, and translation drafts across English and many other languages. Deployment should add teacher review, terminology constraints, privacy filtering, and factuality checks.
Coding assistance
Engineering teams can request code explanations, simple bug fixes, and commented examples, such as the report’s recursive DFS repair. Safe integration requires sandbox execution, tests, code review, and secret isolation.
Long-term Vision
Affordable global assistants
Efficient PaLM 2 variants could enable real-time multilingual support on phones, in customer service, and in public services. Obstacles include low-resource quality, bias, regulation, latency, and continuous model updates.
Abstract
We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.