GptGet

Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

TL;DR

Sailor2 achieves a 50-50 win rate against GPT-4o in SEA languages, supporting 13 languages.

cs.CL 🔴 Advanced 2025-02-19 11 views
Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang Ziqi Jin Zichen Liu Tongyao Zhu Cunxiao Du Penghui Yang Haonan Wang Jiaheng Liu Yongchi Zhao Xiachong Feng Xin Mao Man Tsung Yeung Kunat Pipatanakul Fajri Koto Min Si Thu Hynek Kydlíček Zeyi Liu Qunshu Lin Sittipong Sripaisarnmongkol Kridtaphad Sae-Khow Nirattisai Thongchim Taechawat Konkaew Narong Borijindargoon Anh Dao Matichon Maneegard Phakphum Artkaew Zheng-Xin Yong Quan Nguyen Wannaphong Phatthiyaphaibun Hoang H. Tran Mike Zhang Shiqi Chen Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin
AI Reader Arxiv Page Download PDF
multilingual models SEA languages continual pre-training model expansion data cleaning

Key Findings

Methodology

Sailor2 employs a two-stage continual pre-training strategy, with the first stage using balanced data mixture and the second stage annealing with high-quality data. It supports 13 SEA languages while retaining proficiency in Chinese and English. Training is accelerated using Megatron-LM optimizations and Zero Bubble Pipeline Parallelism.

Key Results

  • Sailor2-20B achieves a 50-50 win rate against GPT-4o in SEA languages, significantly improving SEA language processing capabilities.
  • On SEA-WildBench, Sailor2-20B outperforms models like Llama3.1-70B.
  • Data cleaning and optimization lead to excellent SEA language perplexity performance.

Significance

Sailor2 fills the gap in SEA language models, enhancing NLP capabilities for low-resource languages. It provides new insights for multilingual model development, especially in data cleaning and model expansion.

Technical Contribution

Sailor2 innovates in model expansion and data mixture strategies, employing two-stage pre-training and instruction tuning, significantly improving performance on low-resource languages. It offers a comprehensive guide for multilingual LLM development.

Novelty

Sailor2 is the first to achieve performance on par with GPT-4o in SEA languages, using innovative data mixture and model expansion strategies.

Limitations

  • Performance bottlenecks remain in low-resource languages, requiring further optimization.
  • High training costs and resource consumption.

Future Work

Future work will continue to optimize the Sailor series, exploring support for more low-resource languages and improving model efficiency and performance.

AI Executive Summary

Sailor2 is a multilingual model targeting SEA languages, supporting 13 languages and achieving performance on par with GPT-4o in SEA languages. Through two-stage continual pre-training and instruction tuning, Sailor2 significantly enhances processing capabilities for low-resource languages. Its release not only fills the gap in SEA language models but also provides new insights for multilingual model development, particularly in data cleaning and model expansion. Despite performance bottlenecks in low-resource languages, Sailor2's technical contributions and innovations offer important references for future research.

Deep Analysis

Background

In recent years, multilingual models have made significant progress in NLP, but SEA languages have been largely neglected due to resource scarcity. The Sailor2 project aims to fill this gap by developing open-source models supporting SEA languages.

Core Problem

The diversity and resource scarcity of SEA languages make developing high-performance multilingual models challenging. Existing models perform poorly on these languages, necessitating new methods to enhance their processing capabilities.

Innovation

Sailor2 employs a two-stage continual pre-training strategy, combining data cleaning and model expansion to significantly improve performance on SEA languages. By introducing high-quality data and optimized training methods, Sailor2 achieves breakthroughs in low-resource languages.

Methodology

  • �� Data Cleaning: Utilizes a six-layer filtering mechanism to remove redundant data.
  • �� Model Expansion: Expands model capacity based on Qwen2.5.
  • �� Two-Stage Pre-Training: Stage one uses balanced data mixture, stage two anneals with high-quality data.
  • �� Instruction Tuning: Fine-tunes using the SEA-UltraChat dataset.

Experiments

Experiments were conducted using datasets like SEA-WildBench to evaluate Sailor2's performance compared to other models in SEA languages. Comparative analysis verified Sailor2's advantages in low-resource languages.

Results

Sailor2 achieves performance on par with GPT-4o in SEA languages, especially excelling on SEA-WildBench. Data cleaning and optimization lead to excellent SEA language perplexity performance.

Applications

Sailor2 can be used for translation, text generation, and question-answering systems in SEA languages, particularly suitable for applications requiring multilingual support.

Limitations & Outlook

Despite significant progress, Sailor2 faces performance bottlenecks in low-resource languages, requiring further optimization.

Plain Language Accessible to non-experts

Imagine you're in a multilingual library, and Sailor2 is like a super translator who can quickly and accurately translate books in different languages into one you understand. By continuously learning and optimizing, it handles various SEA languages like an experienced librarian who knows how to find and organize the information you need.

ELI14 Explained like you're 14

Imagine you have a super smart friend named Sailor2 who can speak 13 SEA languages! Whether it's Thai, Vietnamese, or Indonesian, he can help you translate and understand. Just like in a game where you need a powerful character to defeat enemies, Sailor2 is the one who can help you win the match!

Glossary

Sailor2

A multilingual model supporting SEA languages, capable of processing 13 languages.

Used for NLP tasks in SEA languages.

Continual Pre-Training

A method to enhance model performance through multi-stage training.

Used to improve Sailor2's performance on low-resource languages.

Data Cleaning

The process of removing redundant and useless information from datasets.

Improves the quality of training data for Sailor2.

Instruction Tuning

Fine-tuning a model using a specific task dataset.

Enhances Sailor2's performance on specific tasks.

SEA-WildBench

A dataset for evaluating SEA language model performance.

Used to assess Sailor2's performance.

Open Questions Unanswered questions from this research

  • 1 How to further improve performance on low-resource languages?
  • 2 How to effectively balance performance across different languages in multilingual models?

Applications

Immediate Applications

SEA Language Translation

Sailor2 can be used to develop efficient SEA language translation tools, aiding cross-language communication.

Long-term Vision

Global Multilingual Support

By optimizing and expanding, Sailor2 could become a global multilingual platform supporting more languages.

Abstract

Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on Qwen2.5, Sailor2 undergoes continuous pre-training on 500B tokens (400B SEA-specific and 100B replay tokens) to support 13 SEA languages while retaining proficiency in Chinese and English. Sailor2-20B model achieves a 50-50 win rate against GPT-4o across SEA languages. We also deliver a comprehensive cookbook on how to develop the multilingual model in an efficient manner, including five key aspects: data curation, pre-training, post-training, model customization and evaluation. We hope that Sailor2 model (Apache 2.0 license) will drive language development in the SEA region, and Sailor2 cookbook will inspire researchers to build more inclusive LLMs for other under-served languages.

cs.CL cs.AI cs.LG