Scaling Native Multimodal Pre-Training From Scratch
This paper derives power-law scaling laws for from-scratch vision-language pretraining, revealing distinct behaviors for language and multimodal objectives under fixed compute.
Key Findings
Methodology
The study employs IsoFLOP profiles and training curve envelopes to analyze transformer-based vision-language models trained from scratch under fixed compute budgets. By fitting loss curves across various model sizes and token counts, the authors establish that minimal loss follows predictable compute laws. They further model how data composition (multimodal ratio r) influences the scaling exponents for language and multimodal objectives, resulting in a Pareto frontier of optimal configurations. Experiments use a mixture of 250B text tokens and 75B multimodal tokens, with models built on a decoder-only MoE architecture trained on web, books, and academic data. The dual estimation methods validate the scaling laws' robustness.
Key Results
- The minimal loss adheres to a power-law compute law, with model size N and token count D scaling as N ∝ C^a and D ∝ C^b, where a + b ≈ 1. The language objective's scaling exponent a remains nearly invariant across data ratios (~0.67), indicating stable language learning. Conversely, the multimodal objective's exponent decreases from 0.66 to 0.63 as r increases, showing high sensitivity to data composition. Larger models are more compute-efficient for text-heavy data, shifting the optimal resource allocation towards bigger models at higher r. These laws enable precise configuration of model size and token count for different data mixes.
- The models demonstrate strong cross-modal transfer, notably improving pure-text spatial reasoning by over 20%, and enabling multimodal in-context learning. The findings confirm that from-scratch pretraining can achieve deep modality integration without degrading core language skills, and that scaling laws guide efficient resource utilization across tasks.
- Overall, the research provides a theoretical framework for predictably scaling multimodal foundation models, balancing data composition, model size, and token count to optimize performance while reducing waste.
Significance
This work advances the understanding of how to scale native multimodal models from scratch, addressing a critical gap in the field. By establishing clear, quantitative scaling laws, it enables researchers and practitioners to design models that maximize efficiency and transferability. The insights into data composition effects inform better resource allocation strategies, crucial for deploying large models in real-world scenarios. The demonstrated cross-modal transfer and improved spatial reasoning suggest that such models can serve as more versatile and robust AI systems, capable of handling complex multimodal tasks with predictable scaling behavior. This foundational understanding paves the way for future large-scale multimodal AI, with broad implications for industry and academia.
Technical Contribution
The paper introduces a systematic framework for analyzing from-scratch vision-language model scaling laws, employing two independent estimators—IsoFLOP profiles and training curve envelopes—to validate the power-law relationships. It models the influence of data composition on the scaling exponents, deriving a Pareto frontier that guides optimal resource allocation. The work extends existing scaling law theories (e.g., Kaplan et al., 2020) to the multimodal domain, revealing that language and multimodal objectives follow distinct scaling behaviors. The architecture employs a decoder-only MoE model trained on a large, mixed dataset, demonstrating that these laws hold across different data ratios. The methodology allows precise prediction of model configurations for various compute budgets, significantly advancing the theoretical understanding of multimodal model scaling.
Novelty
This research is the first to systematically characterize the scaling laws of native multimodal pretraining from scratch, emphasizing the distinct behaviors of language and multimodal objectives. It introduces a data composition-dependent model for scaling exponents, revealing that multimodal data ratios critically influence optimal model size and token count. Unlike prior work that mainly focused on unimodal models or post-fusion architectures, this study explicitly models the from-scratch training paradigm, providing a unified theoretical framework. The dual-estimator approach enhances robustness, and the derivation of an explicit Pareto frontier offers practical guidance for resource allocation, marking a significant step forward in multimodal AI theory.
Limitations
- The analysis relies on training loss as a proxy for generalization, which may not fully capture real-world performance. The experiments are conducted on specific datasets and architecture (MoE decoder), limiting the generalizability to other models or data domains. The high computational cost restricts exploration of even larger scales or more diverse data compositions. Additionally, the current framework does not explicitly incorporate model robustness or efficiency metrics beyond loss minimization, leaving room for further refinement.
Future Work
Future research should explore broader architectures, including encoder-based models, and validate scaling laws across diverse datasets and tasks. Incorporating generalization metrics and robustness measures will refine resource allocation strategies. Extending the framework to multi-task or continual learning settings could enhance model versatility. Moreover, integrating hardware-aware optimization and model compression techniques will facilitate deployment in resource-constrained environments, accelerating real-world applications of large-scale multimodal AI.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在准备一场盛大的派对，你需要安排座位、食物和娱乐。座位越多，食物也得准备得越多，但如果准备太少，大家会饿肚子；准备太多，又会浪费。科学家们发现，安排这些东西的最佳比例其实遵循一些数学规律，就像用公式告诉你每个环节该准备多少一样。不同的派对（比如只请朋友，或者请朋友和家人一起）需要不同的安排。通过研究这些规律，能帮你更聪明地准备派对，让每个人都满意又不浪费。这个研究就像帮我们找到模型训练的最佳“派对”方案，让AI变得更聪明、更高效。
ELI14 Explained like you're 14
想象你在组织一个超级大的学习班，有很多学生（模型）和很多教材（Token）。如果学生太少，学不了复杂的东西；如果太多，资源就浪费了。科学家们发现，学生和教材的数量其实遵循一些数学规律，就像用比例来安排座位和教材一样。不同的学习任务（比如只学语言，或者学图片和语言结合）对学生和教材的需求也不一样。有趣的是，学语言时，学生的大小几乎不用考虑教材比例，但学图片和语言结合时，就需要更大规模的学生和更多教材。研究帮助我们知道怎么合理分配资源，让学习既高效又不浪费。
Abstract
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.