IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Recent NLP breakthroughs, such as BERT and RoBERTa, have revolutionized language understanding, primarily in high-resource languages like English and Chinese. Multilingual models like mBERT and XLM-R extended these advances to many languages but often underperform in low-resource settings. Indonesian, despite its large internet user base, remains underrepresented due to limited datasets and resources. Existing datasets are scattered, with inconsistent annotations and lack of standard benchmarks, impeding progress. Prior efforts to develop Indonesian-specific models have been limited in scope and scale. The necessity for a unified, large-scale resource and evaluation framework is urgent to unlock NLP's potential for Indonesian, enabling applications like sentiment analysis, named entity recognition, and question answering to reach practical maturity.

The core challenge lies in the scarcity of high-quality, large-scale annotated datasets and pre-trained models tailored for Indonesian. Current datasets are fragmented, with inconsistent formats and limited scope, making model training and evaluation unreliable. Additionally, existing benchmarks focus mainly on English or Chinese, leaving Indonesian without a standardized platform for fair comparison. This resource gap hampers the development of accurate, robust NLP systems in Indonesian, affecting both academic research and industry deployment. Overcoming these barriers requires constructing comprehensive datasets, designing effective pretraining strategies, and establishing multi-task benchmarks that reflect real-world applications.

The paper's key innovations include: 1) the creation of IndoNLU, a multi-task benchmark covering 12 tasks across various domains and styles, standardizing evaluation for Indonesian NLP; 2) the assembly of Indo4B, a large-scale (~23GB) corpus from diverse sources, enabling effective self-supervised pretraining; 3) development of IndoBERT and IndoBERT-lite, monolingual transformer models trained on Indo4B, outperforming multilingual models in downstream tasks; 4) comprehensive comparison and analysis of models, demonstrating the benefits of language-specific pretraining. These innovations collectively push Indonesian NLP from resource scarcity toward resource abundance, setting a new standard for low-resource language research.

• �� Data collection: Aggregated Indonesian texts from social media, news, Wikipedia, subtitles, and parallel corpora, forming Indo4B. • Data preprocessing: Cleaned texts, segmented sentences, masked tokens, preserved morphological richness, masked private info in social media data. • Model training: Used transformer architectures (BERT, ALBERT variants), trained with MLM and sentence prediction objectives, with hyperparameters tuned for size and data scale. • Task design: Developed 12 tasks, including classification, sequence labeling, and sentence pair tasks, with standardized splits. • Evaluation: Employed macro F1-score, early stopping based on validation, hyperparameter optimization, and ablation studies to assess model performance. • Baseline comparison: Benchmarked against mBERT, XLM-R, and scratch models, analyzing parameter effects and data impact.

Experiments involved training multiple models (IndoBERT, IndoBERT-lite, mBERT, XLM-R) on Indo4B, with hyperparameters like learning rate (1e-5 to 4e-5), batch size (8-256), and steps (up to 1 million). Data splits were standardized for reproducibility. Fine-tuning was performed for each task, optimizing hyperparameters via validation. Performance was measured using macro F1 scores across tasks, with results showing IndoBERT-LARGE achieving the highest average scores. Ablation studies confirmed that larger models and more extensive pretraining data improve performance. Cross-task analysis demonstrated the models' robustness and generalization capabilities, validating the effectiveness of the resource and training strategies.

IndoBERT-LARGE achieved an average F1 of 81.21% on classification tasks, outperforming XLM-R-LARGE (79.76%). In sequence labeling, IndoBERT-LARGE scored 81.26%, surpassing multilingual models. The models trained on Indo4B significantly outperformed those trained on smaller or less diverse datasets, confirming the importance of large-scale native corpora. The multi-task evaluation showed consistent improvements, especially in entity recognition and sentiment analysis. The results highlight the advantage of language-specific pretraining, with performance gains more pronounced as model size increases. These findings demonstrate the effectiveness of the proposed resources and training framework.

The benchmark and models can be directly applied to Indonesian sentiment analysis, named entity recognition, and question answering systems, improving accuracy and robustness. Industries such as social media monitoring, customer service, and content moderation will benefit from these tools. Academic researchers can use the standardized benchmark to evaluate new models, fostering innovation. The models also serve as foundational components for developing multilingual or cross-lingual NLP systems, enabling broader language inclusivity. Long-term, these resources can support the development of intelligent virtual assistants, automated translation, and inclusive AI solutions tailored to Indonesian speakers.

High computational costs for training large models limit accessibility for smaller labs. Some tasks suffer from limited annotated data, affecting generalization. Handling informal slang and rapidly evolving language remains challenging, requiring continuous data updates. The models may also struggle with domain-specific jargon outside the training scope. Future work should focus on model compression, domain adaptation, and continual learning to address these issues.