Dango: A Strictly L1-Only Large Language Model for Studying Second Language Acquisition
Dango is a 1.8B-parameter L1-only model, using filtering to reduce L2 contamination, for studying L1-to-L2 transfer in SLA.
Key Findings
Methodology
This study combines data filtering and fine-tuning. First, character whitelist and line-level blocklist filters are applied to Japanese corpus to minimize English contamination, training a pure Japanese pretrained model. Then, multilingual LLM-generated English learning data is used for fine-tuning, simulating SLA. The 1.8B model trained on 100B tokens shows effective transfer features. Evaluation includes linguistic assessments and transfer analysis, demonstrating human-like English production, outperforming unfiltered and multilingual baselines.
Key Results
- Dango achieves 94% accuracy on JBLiMPP Japanese grammatical tasks, close to human performance. On MultiBLiMP, error rates are below 0.02, with feature distributions similar to learners. BLEU scores across levels reach up to 37.2, indicating strong transfer. Filtering effectively reduces English contamination, balancing multilingual abilities. The model produces long, coherent English sentences with transfer features matching human learners, validated by distributional similarity metrics.
- Filtering significantly suppresses English exposure, enabling balanced multilingual performance. Fine-tuning on generated textbook-style data enhances transfer features, with BLEU gains of +12.5 when expanding training data from A1–B1 to A1–C2 levels. The model generalizes well across difficulty levels and datasets, showing strong potential as an SLA simulation tool.
- Analysis confirms that Dango’s English outputs closely resemble human learner patterns, with transfer features such as tense, agreement, and lexical collocations. The model’s ability to produce natural, long English sentences demonstrates its utility for SLA research and educational applications.
Significance
This work addresses the challenge of L2 contamination in large-scale pretraining, providing a controlled environment for SLA modeling. By effectively filtering data and leveraging generated教材，Dango offers a realistic simulation of human second language transfer, bridging cognitive science and NLP. Its open-source release supports research in language learning, virtual teaching, and cross-linguistic transfer analysis, promising to improve SLA understanding and language education tools. The approach sets a new standard for uncontaminated, scalable language modeling for SLA studies, enabling more accurate and interpretable experiments.
Technical Contribution
The paper introduces a novel filtering pipeline combining whitelist and blocklist strategies at character and line levels, significantly reducing English contamination in web-crawled Japanese corpora. The model architecture follows Llama-2 style, scaled to 1.8B parameters, trained on 100B tokens, surpassing prior small models. Fine-tuning on LLM-generated textbooks enhances transfer capabilities. The evaluation framework includes linguistic feature analysis, cross-dataset generalization, and distributional similarity metrics, establishing a comprehensive methodology for SLA modeling with large language models. This work demonstrates how controlled data curation and targeted fine-tuning can produce models capable of human-like L2 transfer.
Novelty
This is the first systematic effort to address L2 contamination in large-scale pretraining for SLA research. It combines a rigorous filtering method with a curriculum-style fine-tuning on LLM-generated educational data, enabling the model to exhibit human-like transfer features. Unlike previous models trained on mixed multilingual data, Dango maintains a pure L1 foundation and demonstrates effective L2 transfer, filling a critical gap in computational SLA. Its scalable architecture and evaluation methodology set new standards for future research.
Limitations
- Despite filtering, some residual English contamination remains, which may influence transfer features subtly. Further refinement of filtering algorithms is needed.
- High computational costs for training and fine-tuning limit accessibility, especially for smaller research groups.
- The model’s transfer abilities are primarily tested on written data; real-time interactive or conversational capabilities require further development. Additionally, the model’s performance on complex, context-dependent language tasks remains limited.
Future Work
Future directions include developing more sophisticated multilingual filtering techniques, integrating real human learner data for curriculum refinement, and expanding the model’s capabilities to conversational SLA scenarios. Exploring multi-modal inputs and structural learning could enhance understanding of complex transfer phenomena. Additionally, applying this framework to other language pairs and low-resource languages will broaden its impact, ultimately aiming to create versatile, scalable SLA simulation tools for both research and educational practice.
AI Executive Summary
The rapid expansion of large language models has opened new avenues for studying second language acquisition (SLA) through computational means. However, existing models often suffer from contamination of L2 data within predominantly L1 corpora, which hampers their ability to accurately simulate human-like transfer phenomena. Addressing this challenge, the authors introduce Dango, a 1.8-billion-parameter decoder-only model trained on carefully filtered Japanese data to minimize English exposure. This filtering process combines character whitelist and line-level blocklists, significantly reducing English contamination while preserving essential linguistic signals.
Building on this foundation, the researchers generate textbook-style English learning data using a multilingual LLM, covering various proficiency levels based on CEFR-J standards. Fine-tuning Dango on this synthesized data enables it to emulate the SLA process, producing English sentences that mirror human learners’ transfer patterns. Extensive evaluations, including linguistic feature analysis, distributional similarity metrics, and cross-dataset generalization, demonstrate that Dango exhibits human-like transfer effects, outperforming unfiltered and multilingual baselines.
The significance of this work lies in its ability to provide a controlled, scalable platform for SLA research, bridging cognitive science and NLP. By effectively controlling data contamination, Dango offers a realistic simulation of second language learning, with applications spanning virtual teaching, language assessment, and transfer analysis. Its open-source release ensures broad accessibility, fostering further research and development in SLA modeling.
Looking ahead, the authors plan to refine filtering techniques, incorporate real learner data, and extend the framework to other languages and modalities. This work sets a new standard for uncontaminated, large-scale language modeling tailored to SLA, promising to deepen our understanding of language transfer mechanisms and improve language education technologies.
Deep Dive
Glossary
L1 (First Language)
The native language acquired during childhood, fundamental for language learning research. In this paper, it refers to Japanese, the primary pretraining language.
Used to describe the initial language model training phase.
L2 (Second Language)
A language learned after the first language, often through formal education or immersion. Here, English is modeled as the second language.
Focus of transfer effect analysis and fine-tuning.
Filtering Pipeline (过滤流程)
一套字符和行级过滤机制，用于在预训练语料中减少英语污染，确保模型专注于L1学习。
用于预训练数据的净化，提升迁移模拟的真实性。
CEFR-J (欧洲共同语言参考标准-Japanese)
基于欧洲共同语言参考标准，为日本学习者设计的英语能力分级体系，用于教材和评估。
生成教材式英语学习数据的依据。
Transfer Effect (迁移效应)
学习者在第二语言中表现出受第一语言影响的语言特征，包括语法、词汇和表达习惯。
模型输出中的迁移特征分析。
Open Questions Unanswered questions from this research
- 1 如何进一步减少过滤后残留的英语污染，确保迁移特征的纯粹性。
- 2 模型在真实交互场景中的迁移表现及其适应性研究。
- 3 多语种环境下的迁移机制差异及其建模方法。
Applications
Immediate Applications
虚拟二语教师
利用Dango模型为学习者提供个性化英语辅导，模拟真实学习场景，提升学习效果。
迁移特征研究工具
作为研究二语迁移机制的实验平台，帮助学者分析不同语言背景下的迁移规律。
Long-term Vision
智能语言学习系统
结合模型的迁移能力，开发能自主调整教学内容的智能平台，推动个性化教育普及。
Abstract
We introduce Dango, a 1.8B-parameter large language model designed for controlled studies of L1-to-L2 (Japanese-to-English) transfer in second language acquisition (SLA). While previous studies have explored SLA in language models, they have predominantly relied on smaller or non-decoder models, limiting their ability to generate open-ended text and reducing their suitability as practical L2 simulators. We identify a key challenge when scaling models to this size: L2 contamination within the "monolingual" pretraining corpus used for L1 acquisition. To address this, we propose a filtering method to reduce premature exposure to English while preserving realistic, minimal exposure. We then fine-tune the model on LLM-generated L2-learning lessons to simulate the L2 acquisition process. Our evaluations confirm that Dango develops human-like L2 production patterns, outperforming both unfiltered and standard multilingual baselines. We release the model, data, and code to facilitate reproducible computational SLA research and learner-facing applications.