KLUE: Korean Language Understanding Evaluation
KLUE constructs 8 Korean NLU tasks, using from-scratch data collection and pretrained models KLUE-BERT and KLUE-RoBERTa, outperforming multilingual and open-source baselines.
Key Findings
Methodology
This work collects diverse, publicly available corpora, designs 8 tasks reflecting Korean linguistic features, and employs strict annotation protocols combining automatic filtering and manual validation. Pretraining of KLUE-BERT and KLUE-RoBERTa models uses multi-task learning with Transformer architectures, optimized via AdamW, integrating BPE and morpheme-level tokenization to enhance morpheme-level tasks. Evaluation metrics include F1, accuracy, Pearson correlation, UAS, and LAS, ensuring comprehensive assessment. The models are fine-tuned per task, demonstrating strong transferability and scalability.
Key Results
- KLUE-RoBERTa-large achieves an average F1 of 87.3% across tasks, surpassing multilingual and open-source models by over 5%, especially excelling in relation extraction and machine reading comprehension, with significant performance gains.
- Removing personally identifiable information from training data results in negligible performance drops, indicating privacy preservation does not compromise NLU capabilities. The combined BPE and morpheme pre-tokenization significantly improves morpheme-level tagging and detection accuracy.
- Larger models like RoBERTa-Large outperform smaller counterparts, confirming the importance of model scale. The models generalize well across tasks, with consistent improvements over baselines, validating the effectiveness of the training strategies.
Significance
This benchmark addresses the lack of standardized Korean NLU evaluation, providing a comprehensive, ethically designed dataset and models. It accelerates research by offering open resources, fostering fair comparison, and guiding future model development. The focus on privacy and bias mitigation sets industry standards. The work bridges the gap between academic research and practical applications, enabling robust Korean NLP systems for industry and societal needs.
Technical Contribution
The study introduces a full pipeline for Korean NLU benchmark creation: from multi-source data collection with open licenses, to task-specific annotation respecting Korean morphology, to training dedicated Transformer-based models with multi-task fine-tuning. The integration of morpheme-aware tokenization and privacy-preserving techniques distinguishes this work from prior multilingual or translation-based benchmarks. The models demonstrate superior performance, validating the proposed methodologies.
Novelty
This is the first comprehensive Korean NLU benchmark built from ground-up data collection, annotation, and dedicated pretraining. The innovative use of morpheme-level tokenization combined with BPE improves performance on morphologically rich tasks. The models trained outperform existing multilingual models, establishing new state-of-the-art results in Korean NLP, and setting a standard for low-resource language benchmarks.
Limitations
- Data filtering may exclude edge cases, potentially limiting model robustness in rare or complex scenarios.
- Pretraining large models incurs high computational costs, restricting widespread reproduction.
- Performance on extremely informal or dialectal Korean remains limited; future work should incorporate more diverse data sources.
Future Work
Future directions include expanding to more NLP tasks like sentiment analysis and dialogue generation, exploring multimodal learning, and enhancing privacy-preserving techniques. Cross-lingual transfer and low-resource adaptation are also promising avenues. Improving model efficiency and robustness, especially on dialectal and informal Korean, will be prioritized to broaden real-world applicability.
AI Executive Summary
KLUE (Korean Language Understanding Evaluation) is the first comprehensive benchmark designed to evaluate Korean NLP systems across diverse tasks. Comprising eight tasks—topic classification, semantic similarity, natural language inference, named entity recognition, relation extraction, dependency parsing, machine reading comprehension, and dialogue state tracking—KLUE captures the multifaceted nature of Korean language understanding.
The benchmark was built from scratch, sourcing data from publicly available corpora like news, Wikipedia, and reviews, with strict adherence to copyright and ethical standards. Data annotation involved multiple rounds of manual and automatic filtering to eliminate biases, toxic content, and PII, ensuring high-quality, ethically sound datasets. This meticulous process guarantees that the benchmark reflects real-world language use while respecting societal norms.
To facilitate research, the authors pretrained two models—KLUE-BERT and KLUE-RoBERTa—using Transformer architectures tailored for Korean morphology, integrating morpheme-aware tokenization strategies. These models were fine-tuned on each task, achieving state-of-the-art results, with KLUE-RoBERTa-large attaining an average F1 score of 87.3%, outperforming existing multilingual and open-source models. Notably, privacy-preserving measures had minimal impact on performance, demonstrating that privacy and high NLU capability can coexist.
This work significantly advances Korean NLP by providing a standardized, open benchmark and robust models, fostering fair comparison and rapid progress. Its comprehensive documentation and open resources lower barriers for researchers and industry practitioners. Future work will expand task coverage, incorporate multimodal data, and refine privacy techniques, ensuring KLUE remains a vital tool for Korean NLP development and societal integration.
Deep Analysis
Background
近年来，预训练语言模型如BERT、GPT系列在自然语言处理领域取得突破，推动了多语言、多任务模型的发展。英语作为主导语言，已有诸如GLUE、SuperGLUE等成熟基准体系，但韩语缺乏统一的评测平台。现有韩语数据集多为特定任务或小规模，难以全面衡量模型能力。随着韩语应用场景的多样化，亟需构建系统化、标准化的评估体系，促进模型性能提升和应用落地。此前的研究多依赖翻译或少量公开数据，存在偏差和局限性，亟待从数据采集、标注和模型设计全方位优化。
Core Problem
韩语作为一种黏着语，具有丰富的形态变化和复杂的句法结构，给NLU任务带来挑战。现有数据集缺乏多样性和规模，难以支撑深度模型的训练和评估。缺少统一的基准体系，使得不同研究成果难以比较，限制了技术的快速发展。此外，隐私和偏见问题在数据采集和模型训练中尤为突出，亟需系统性解决。综上，韩语NLU的评估体系亟待建立，既要保证数据质量，又要兼顾伦理责任。
Innovation
本研究的创新点主要体现在：1）自主采集多源公开语料，确保数据无版权限制，涵盖多样场景；2）设计符合韩语语法特点的标注协议，提升标注准确性；3）提出结合形态学特性的BPE与预切分技术，优化 morpheme-level任务表现；4）开发专属预训练模型KLUE-BERT和KLUE-RoBERTa，采用多任务微调，显著优于多语模型。此体系实现了从数据到模型的全流程自主创新，为韩语NLU提供了系统性解决方案。
Methodology
- �� 数据采集：从新闻、维基、评论等公开语料库筛选，自动过滤偏见、毒性和个人信息，确保合法合规。• 标注协议：结合韩语形态学特性，设计任务特定的标注指南，反复校验以确保一致性。• 模型预训练：基于Transformer架构，训练KLUE-BERT和KLUE-RoBERTa，采用多任务学习策略优化多任务性能。• 微调策略：在每个任务上进行微调，调整学习率、批次大小等超参数，确保模型适应不同任务需求。• 评估指标：采用F1、准确率、Pearson相关系数、UAS、LAS等指标，全面衡量模型能力。
Experiments
实验设计包括：选择多源公开语料，划分训练、验证、测试集，确保平衡。使用多任务微调策略，比较不同模型（如多语模型、韩语专属模型）在各任务上的性能。通过消除个人信息，验证隐私保护对性能影响。采用多种评估指标，分析模型在不同任务中的优势与不足。进行消融实验，验证BPE与形态素预切分的效果。最后，将模型应用于实际场景，验证其泛化能力。
Results
KLUE-RoBERTa-large在所有任务中表现优异，平均F1达87.3%，比多语模型提升5%以上。在关系抽取和阅读理解任务中，性能提升尤为明显，达6-8%。去除个人识别信息后，模型性能变化极小，验证隐私保护的有效性。结合BPE和形态素预切分的策略，提升了命名实体识别和关系抽取的准确率，特别在 morpheme-level任务中表现优越。模型规模与性能呈正相关，验证了大模型的优势。
Applications
该基准体系可广泛应用于韩语智能客服、信息检索、内容分析等场景。企业可利用预训练模型快速部署多任务系统，提升效率。学术界可借助KLUE评估新模型，推动韩语NLP技术发展。未来还可结合多模态信息，拓展到语音、图像理解，满足更复杂的应用需求。
Limitations & Outlook
数据采集依赖自动过滤，可能遗漏少数边缘案例，影响模型泛化。模型训练成本高，尤其在大规模预训练阶段，限制部分研究者复现。部分任务在极端语境或少数群体语料中表现仍需提升，未来需多样化数据源。模型在处理长句或复杂句结构时仍有待优化，此外，偏见和隐私保护技术仍需持续改进。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂的任务是把各种原料变成成品。以前，工厂用不同的机器处理不同的原料，效率不高。现在，研究人员设计了一台特别的机器（模型），可以用一种方法处理各种原料（不同任务），还可以学习新技巧（多任务学习）。他们还收集了大量的原料（语料库），确保没有偏见或隐私问题。经过训练，这台机器变得非常聪明，能理解和处理韩语中的各种复杂句子，就像一个多才多艺的工厂助手。这个系统让韩语的AI变得更强大，也为未来的研究提供了标准。
ELI14 Explained like you're 14
想象你在学校里，有一台超级智能的机器人老师，它可以帮你理解各种不同的科目，比如语文、数学、科学。以前，每个科目都要专门学一套教材，机器人老师也得专门训练。现在，科学家们设计了一套特别的学习方法，让机器人用一种通用的方式学会所有科目。它们还收集了很多公开的资料，比如新闻、百科全书、评论，然后教给机器人。通过反复练习，这个机器人变得非常聪明，能理解复杂的句子、找出文章中的人物和地点、理解句子之间的关系。它还能帮你回答问题，甚至理解你说的话是不是在表达同样的意思。这个研究让机器人变得更聪明，也让我们更容易用它来帮忙解决实际问题，比如客服、内容筛选等。未来，这样的机器人会变得更厉害，能理解更多场景，帮助我们生活得更方便。
Abstract
We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.