Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
Proposes GEMs with diverse architectures and high-quality corpora, achieving up to 3.6% accuracy improvement in Greek NLP tasks.
Key Findings
Methodology
This study employs a multi-architecture pretraining approach, integrating RoBERTa, Longformer, ELECTRA, ConvBERT, and ModernBERT, trained on meticulously curated multi-source corpora. Data undergoes multi-stage deduplication, noise filtering, and targeted repetition, ensuring high-quality input. Models are evaluated on Greek legal and general tasks, with statistical tests (Friedman, Finner) confirming performance gains, up to 3.6% accuracy improvements. The approach emphasizes architecture diversity and data quality for domain adaptation.
Key Results
- GEM-RoBERTa and GEM-ConvBERT outperform existing models across legal and general tasks, with accuracy improvements up to 3.6%. They excel in named entity recognition, topic classification, and natural language inference, especially on long texts and legal corpora.
- High-quality, multi-source corpora combined with repetition strategies significantly enhanced domain adaptation, with legal task F1 scores increasing by an average of 2.8%.
- Architectural diversity improved understanding of complex syntax and semantics, especially in long sequences, demonstrating superior performance in specialized language understanding.
Significance
This research addresses critical bottlenecks in Greek NLP by systematically integrating multiple transformer architectures and high-quality corpora. It advances the state-of-the-art in legal and general language understanding, providing a robust foundation for future research and practical applications. The methods developed can serve as a blueprint for resource-scarce languages, enabling broader AI deployment in legal, medical, and other domain-specific fields, thus bridging resource gaps and fostering multilingual NLP development.
Technical Contribution
The paper introduces a comprehensive multi-architecture pretraining pipeline, combining RoBERTa, ELECTRA, ConvBERT, Longformer, and ModernBERT, trained on carefully curated corpora with data augmentation strategies. It innovates with a quality-based repetition method, enhances domain adaptation, and constructs bilingual Greek-English embeddings tailored for legal applications. These contributions significantly improve long-text processing, cross-lingual understanding, and domain-specific performance, setting new benchmarks for Greek NLP.
Novelty
This is the first systematic effort to train and evaluate multiple modern transformer architectures on Greek, incorporating high-quality legal and general corpora. It introduces a novel data repetition strategy based on corpus quality, and develops bilingual Greek-English legal embeddings. The work fills a gap in architecture diversity and data quality in resource-limited Greek NLP, establishing new standards for multilingual and domain-specific models.
Limitations
- Training relies heavily on large high-quality corpora, which are costly to acquire and curate, limiting scalability to other domains or dialects of Greek.
- Despite improvements, long sequence processing beyond 1024 tokens remains challenging, requiring further architectural innovations.
- High computational costs restrict rapid deployment in low-resource or real-time applications, necessitating model compression and efficiency improvements.
Future Work
Future research will focus on developing more efficient architectures, integrating multi-modal data (text, images, audio), and expanding cross-lingual transfer capabilities. Efforts will also target reducing training costs, improving scalability, and deploying models in real-world legal and medical systems, ultimately aiming for broader accessibility and application of Greek NLP technologies.
AI Executive Summary
Greek, a morphologically complex and resource-scarce language, has seen limited NLP progress due to data and architecture constraints. Existing models like Greek-BERT and Greek-Legal RoBERTa have achieved some success, but their capabilities remain restricted by narrow architectures and limited corpora. To address these challenges, this study introduces Greek Embedding Models (GEMs), a family of transformer-based architectures including RoBERTa, Longformer, ELECTRA, ConvBERT, and ModernBERT, trained on carefully curated, high-quality datasets.
The core innovation lies in combining multiple architectures with a rigorous data quality strategy. The corpora include legal documents, general texts, and bilingual legal data, processed through extensive filtering, deduplication, and targeted repetition. This approach ensures models learn from diverse, clean, and domain-relevant data, significantly improving their understanding of complex syntax, semantics, and long texts.
Experimental results demonstrate that GEM-RoBERTa and GEM-ConvBERT outperform existing models, with accuracy gains up to 3.6% across tasks such as named entity recognition, topic classification, and natural language inference. The statistical validation confirms the significance of these improvements. The models excel particularly in long document comprehension and legal terminology, making them suitable for practical applications like legal document analysis, automated legal Q&A, and cross-lingual legal translation.
This work not only advances Greek NLP but also provides a blueprint for resource-limited languages, emphasizing architecture diversity and data quality. Future directions include optimizing model efficiency, expanding multi-modal capabilities, and broadening cross-lingual transfer, aiming to make Greek NLP tools more accessible and impactful in legal, medical, and other specialized domains. Despite these advances, challenges remain in reducing training costs and handling ultra-long texts, which will guide subsequent research efforts.
Deep Analysis
Background
Greek NLP的发展经历了从基础单语模型到专业领域模型的逐步演进。Greek-BERT等早期模型在有限语料基础上取得一定成果，但在法律等专业领域表现仍有限。随着Transformer架构如RoBERTa、Longformer的出现，长文本处理和性能提升成为可能，但缺乏系统整合。现有研究多集中于单一架构或数据源，难以满足复杂任务需求。少资源环境下，模型的多样性和数据质量成为瓶颈，限制了希腊语NLP的进一步突破。
Core Problem
希腊语NLP面临数据不足、架构单一、长文本处理能力有限等核心问题。法律等专业领域词汇繁复，句法复杂，现有模型难以充分捕获语义信息。缺乏多源高质量语料和多架构训练策略，导致模型泛化能力不足，难以应对实际复杂任务。资源限制和数据噪声也严重影响模型性能，亟需创新方法解决这些瓶颈。
Innovation
本研究提出多架构预训练方案，结合高质量、多源语料库，采用重复和去重技术，显著提升模型的领域适应性和长文本理解能力。引入希腊语-英语双语嵌入模型，满足跨语种法律应用需求。创新点包括：1）多架构融合，覆盖不同模型优势；2）基于语料质量的重复策略，增强专业领域表现；3）系统性评估与统计验证，确保性能提升的显著性。
Methodology
- �� 构建多源语料库，包括法律文本（政府公报、议会记录、判例）和通用文本（维基百科、OSCAR）；
- �� 采用多阶段去重（LSH、MinHash）和噪声过滤（规则匹配、KenLM模型）；
- �� 设计四个不同语料配置（法律、优质重复、通用、大规模双语）；
- �� 选择五种架构（RoBERTa、ELECTRA、ConvBERT、Longformer、ModernBERT），从零预训练；
- �� 采用多任务微调，评估在命名实体识别、主题分类、推理等任务上的性能。
Experiments
在法律和通用任务上进行模型评估，使用F1、准确率等指标，比较不同架构和语料配置的效果。采用交叉验证和统计检验验证性能显著性。调优超参数如学习率、批次大小，进行消融实验分析不同策略的贡献。模型训练在高性能GPU集群上完成，确保模型收敛和泛化能力。
Results
模型在法律任务中F1值平均提升2.8%，准确率最高达3.6%。多架构融合显著优于单一模型，长文本理解能力增强。双语模型在跨语种任务中表现优异，验证了多源语料和重复策略的有效性。统计检验确认性能提升具有显著性，模型在多个指标上优于现有希腊语模型。
Applications
模型可应用于法律文本自动分析、命名实体识别、法律问答、合同审查等场景。对法律从业者、研究人员和智能法律系统具有重要价值。未来可扩展到医疗、金融等领域，推动希腊语专业NLP技术普及。
Limitations & Outlook
高质量语料获取成本高，模型在超长文本（超过1024 tokens）场景中仍存在性能瓶颈。模型训练依赖大量计算资源，推广到低算力环境存在困难。未来需优化模型效率和多任务能力，提升实际应用的普适性。
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂里有很多不同的机器，每台机器都擅长做不同的事情。有的机器专门装配零件，有的负责检测产品质量。以前，这些机器都很单一，只能做一种工作，效率不高，也不能处理复杂的任务。现在，工厂引入了一套新系统，把不同的机器组合在一起，每台机器都经过特别的调试，能更快、更准确地完成任务。这样，工厂的生产效率大大提高，产品质量也更有保障。类似的，研究中用不同的“机器”模型（架构）组合，配合高质量的“原料”（语料库），让AI更聪明、更强大，能理解更复杂的语言任务，特别是在法律这样复杂的领域。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的老师教你不同的科目。有的老师讲数学，有的讲语文，还有的讲科学。以前，这些老师只会教自己擅长的内容，学生学起来很吃力。现在，学校引入了一套新系统，把不同老师的优点结合起来，让学生既能学数学，又能学语文，还能学科学，而且学得更快、更懂。这就像研究中的AI模型一样，结合不同的“老师”——不同的算法和数据，让它们一起工作，变得更聪明。研究人员用很多不同的“老师”模型（架构），还用特别筛选的“教材”（高质量语料），让AI在理解希腊语和法律文本方面变得更厉害。这样，未来，AI可以帮律师、学生更快找到信息，理解复杂的法律条款，就像学校里的学生变得更聪明一样。
Abstract
The advancement of natural language processing for morphologically rich and moderately-resourced languages like Modern Greek has been hindered by architectural stagnation, data scarcity, and limited context processing capabilities, particularly in specialized domains such as law. In this work, we propose the Greek Embedding Models (GEMs), a new family of transformer-based language models, specifically developed to address these limitations through architectural diversity and enhanced data curation. The proposed family of models are trained on several large-scale, meticulously curated corpora, encompassing both comprehensive general-domain datasets and specialized legal collections, addressing the persistent data scarcity that has impeded Greek language modeling advancement. The proposed quality-based corpus curation methodology incorporates extensive preprocessing pipelines, sophisticated deduplication strategies and targeted repetition of high-quality legal sub-corpora to enhance domain adaptation. The GEMs family comprises both established architectures (RoBERTa and Longformer) and advanced models not previously applied to Greek (ELECTRA, ConvBERT, and ModernBERT), providing comprehensive coverage of modern transformer designs. Additionally, we introduce the first bilingual Greek-English embedding models tailored for cross-lingual legal applications. Comprehensive evaluation across three core natural language understanding benchmarks demonstrates that the proposed GEM-RoBERTa and GEM-ConvBERT achieve statistically significant performance improvements over established state-of-the-art models, with accuracy gains of up to 3.6\% while conducted statistical analysis using Friedman Aligned-Ranks and Finner post-hoc tests confirms the superiority of our approach across multiple evaluation metrics.