LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval
LEMUR creates a multilingual EU law corpus and fine-tunes state-of-the-art embedding models with contrastive learning to improve legal retrieval accuracy.
Key Findings
Methodology
The paper introduces LEMUR, a large-scale multilingual EU environmental legislation corpus from 24,953 PDFs across 25 languages, with PDF-to-text fidelity assessed via Lexical Content Score (LCS). It fine-tunes three top multilingual embedding models (Qwen3-0.6B, Qwen3-4B, E5-Multilingual) using contrastive objectives in monolingual and bilingual settings, leveraging query-document pairs derived from structured metadata and legislative texts. The models are indexed with ChromaDB for efficient Top-k retrieval. Experiments demonstrate consistent improvements in retrieval accuracy, especially for low-resource languages, with transferability across unseen languages, indicating content-level, language-agnostic representations.
Key Results
- Fine-tuning improves Top-1 accuracy by approximately 5-10% in high-resource languages like English (up to 96.04%) and 15-20% in low-resource languages such as Maltese (80%) and Latvian (87.5%). Cross-lingual transfer results show at least 10% accuracy gains in unseen languages, confirming the models' content-based generalization. The multi-positive contrastive approach enhances alignment across languages, enabling robust long-document retrieval. Results outperform baseline pre-trained models, validating the effectiveness of the proposed methods.
- The models exhibit strong robustness in long-text retrieval, with high recall rates (>90%) across languages. The multi-lingual fine-tuning strategy effectively bridges language gaps, especially benefiting low-resource languages. The experiments also reveal that content-level representations are primarily learned, with minimal language-specific cues, supporting cross-lingual generalization.
- Overall, the research demonstrates that domain-specific contrastive fine-tuning on high-quality multilingual legal corpora significantly advances legal retrieval performance, especially in multilingual and low-resource contexts, setting a new benchmark for legal NLP applications.
Significance
This work addresses the critical challenge of multilingual legal information retrieval by providing a high-quality, domain-specific corpus and an effective contrastive learning framework. It bridges the gap between large-scale pretraining and task-specific fine-tuning, especially for low-resource languages, facilitating more accurate and reliable legal search systems globally. The approach enhances cross-lingual transferability, crucial for international legal cooperation and policy analysis. By establishing a benchmark and methodology for long-text, multilingual legal retrieval, it paves the way for more intelligent legal AI systems capable of understanding complex, multi-language legal documents, thus significantly impacting legal informatics, policy-making, and AI research.
Technical Contribution
The paper introduces the LEMUR corpus with rigorous PDF-to-text quality assessment via LCS, ensuring data reliability. It innovates by applying multi-positive contrastive learning to fine-tune multilingual models, aligning legal content across languages at the semantic level. The models leverage long-text capabilities and are indexed with ChromaDB for scalable retrieval. This combination of high-quality data, novel training strategies, and efficient indexing constitutes a significant technical advancement over existing pretraining or monolingual approaches, enabling robust cross-lingual legal retrieval.
Novelty
LEMUR is the first comprehensive multilingual EU legal corpus derived directly from official PDFs, with systematic quality evaluation. The contrastive fine-tuning strategy employing multi-positive samples across languages is a novel approach that significantly enhances cross-lingual semantic alignment. Unlike prior work focused on classification or monolingual tasks, this study emphasizes long-text, content-level retrieval, addressing a critical gap in multilingual legal NLP. The integration of PDF quality metrics and multi-lingual contrastive learning sets new standards for legal AI research.
Limitations
- Despite high overall performance, the models struggle with very old or poorly formatted PDFs, leading to residual noise affecting retrieval accuracy. The PDF-to-text conversion quality, although assessed, still introduces errors, especially in complex layouts, limiting data fidelity. Computational costs for training and indexing large models across multiple languages are substantial, hindering scalability. The models primarily learn content-level representations, which may overlook nuanced legal language features, and their interpretability remains limited. Future work should focus on improving OCR accuracy, reducing computational demands, and enhancing explainability.
Future Work
Future directions include expanding the corpus to cover more legal domains and languages, integrating knowledge graphs for better semantic understanding, and developing end-to-end pipelines combining PDF extraction, indexing, and retrieval. Exploring domain-adaptive pretraining and multilingual knowledge integration could further boost performance. Additionally, efforts to improve OCR accuracy and model interpretability will be crucial for practical deployment. The ultimate goal is to create scalable, explainable, and universally applicable legal AI systems that support international legal cooperation and policy analysis.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个大型图书馆里，里面有成千上万的法律书籍，都是用不同的语言写成的。过去，你只能用关键词在书架上找书，但有时候找得很慢，也不一定找到最合适的那本。现在，科学家们设计了一个聪明的机器人助手，它可以理解不同语言的法律内容，还能用一种特殊的方式，把不同语言的法律条文变成一种“共同的语言”。这样，无论你用英语、德语还是其他语言问问题，它都能帮你找到对应的法律条文，就像一个懂多国语言的超级图书管理员一样。这个机器人经过特别训练，学会了用内容而不是语言的差异来匹配法律信息，让你在多语环境中也能快速、准确地找到需要的法律资料。这就像是给这个图书馆装上了一台多语的“翻译机”和“搜索引擎”，让法律信息变得更容易获取，也让法律工作变得更高效。
ELI14 Explained like you're 14
想象你在学校图书馆找书，但每本书都用不同的语言写成。以前，你只能用关键词搜索，可能找不到最合适的那本。而现在，有个超级聪明的机器人朋友，它可以理解不同语言的书的内容，还能帮你快速找到你想要的那本书。这个机器人经过特别训练，学会了用不同语言描述同一本书的内容，然后用这些描述找到对应的书。比如，你用英语问，它能用法语、西班牙语甚至马耳他语找到答案。它还知道长长的书可能会有很多章节，能在复杂的排版中找到关键信息。这样，你就不用担心语言不通或找不到书的问题了。这个机器人就像一个多语的超级图书管理员，帮你节省了很多时间，让学习变得更轻松！
Abstract
Large language models (LLMs) are increasingly used to access legal information. Yet, their deployment in multilingual legal settings is constrained by unreliable retrieval and the lack of domain-adapted, open-embedding models. In particular, existing multilingual legal corpora are not designed for semantic retrieval, and PDF-based legislative sources introduce substantial noise due to imperfect text extraction. To address these challenges, we introduce LEMUR, a large-scale multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25 languages. We quantify the fidelity of PDF-to-text conversion by measuring lexical consistency against authoritative HTML versions using the Lexical Content Score (LCS). Building on LEMUR, we fine-tune three state-of-the-art multilingual embedding models using contrastive objectives in both monolingual and bilingual settings, reflecting realistic legal-retrieval scenarios. Experiments across low- and high-resource languages demonstrate that legal-domain fine-tuning consistently improves Top-k retrieval accuracy relative to strong baselines, with particularly pronounced gains for low-resource languages. Cross-lingual evaluations show that these improvements transfer to unseen languages, indicating that fine-tuning primarily enhances language-independent, content-level legal representations rather than language-specific cues. We publish code\footnote{\href{https://github.com/nargesbh/eur_lex}{GitHub Repository}} and data\footnote{\href{https://huggingface.co/datasets/G4KMU/LEMUR}{Hugging Face Dataset}}.