MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer
Introduces MULTI-EURLEX, a multilingual, multi-label legal dataset, and explores zero-shot cross-lingual transfer using models like XLM-ROBERTA and MT5 with adaptation strategies.
Key Findings
Methodology
This study constructs the MULTI-EURLEX dataset, comprising 65,000 European Union legal documents translated into 23 languages, annotated with EUROVOC hierarchical labels. It employs Transformer-based pretrained models such as XLM-ROBERTA and MT5 for multi-task learning, focusing on zero-shot transfer after monolingual fine-tuning. Various adaptation strategies—including partial fine-tuning, adapters, BITFIT, and LNFIT—are evaluated to mitigate catastrophic forgetting of multilingual knowledge. Experiments utilize chronological splits to reflect real-world concept drift, contrasting with random splits to assess performance degradation over time. Evaluation uses mean R-Precision (mRP) to measure transfer robustness across multiple languages and label hierarchies.
Key Results
- Fine-tuning models like XLM-ROBERTA on a single language results in significant performance drops in zero-shot transfer, with average mRP decreasing from 66.6% (pretraining) to 53.7% across 23 languages. Incorporating adaptation strategies, especially adapters, improves performance by up to 6.8 percentage points, maintaining high accuracy in source language while boosting target languages. Temporal splits reveal that concept drift causes model performance to decline progressively over time, with the random split overestimating real-world results. Experiments across label hierarchies show that finer labels increase transfer difficulty, but adaptation strategies remain effective.
- The results demonstrate that adaptation strategies like partial fine-tuning, adapters, BITFIT, and LNFIT significantly enhance cross-lingual transfer, with adapters providing the best overall performance. Models like MT5, which frame NLP tasks as text generation, show strong baseline results but are sensitive to adaptation methods. Notably, parameter-efficient methods such as BITFIT and LNFIT, which update only bias or layer normalization parameters, achieve competitive results with minimal parameter updates. These findings highlight the importance of preserving multilingual knowledge during task-specific fine-tuning.
- Overall, the study provides a comprehensive evaluation of multilingual legal text classification, emphasizing the critical role of adaptation strategies in zero-shot transfer. It underscores the impact of temporal concept drift and offers practical solutions to mitigate it, advancing the development of robust multilingual legal AI systems.
Significance
This research addresses a critical gap in legal NLP by enabling effective cross-lingual transfer across diverse languages and hierarchical labels. The MULTI-EURLEX dataset offers a valuable benchmark for future studies, fostering progress in multilingual legal document understanding. The demonstrated adaptation strategies provide practical tools for deploying legal AI in resource-scarce languages, reducing reliance on extensive labeled data. By highlighting the importance of temporal splits, the study also paves the way for more realistic evaluation protocols that account for concept drift, ensuring models remain effective over time. These advances have the potential to revolutionize legal information retrieval, automated classification, and legal question answering, making legal AI more accessible and scalable globally.
Technical Contribution
The paper's key technical contributions include the creation of the comprehensive MULTI-EURLEX dataset, which covers 23 languages with hierarchical labels, providing a rich resource for cross-lingual legal NLP. It systematically analyzes the catastrophic forgetting problem in multilingual models after monolingual fine-tuning, proposing effective adaptation strategies such as partial layer freezing, adapters, BITFIT, and LNFIT to preserve multilingual knowledge. The study introduces a rigorous experimental framework contrasting chronological and random splits, demonstrating the impact of concept drift. It also benchmarks multiple models, including XLM-ROBERTA and MT5, revealing the strengths and limitations of each in zero-shot transfer scenarios. These insights advance the understanding of multilingual model adaptation and transfer learning in complex legal domains.
Novelty
This work is pioneering in integrating hierarchical multi-label legal classification with cross-lingual transfer, especially under realistic temporal splits that simulate concept drift. Unlike prior studies focusing on general NLP tasks, it emphasizes legal texts' unique challenges, such as label hierarchy and domain-specific language. The innovative use of parameter-efficient adaptation methods (BITFIT, LNFIT) in a multilingual legal context is novel, demonstrating that minimal parameter updates can effectively retain multilingual knowledge. Additionally, the comprehensive evaluation across 23 languages and multiple label granularities distinguishes this work from existing datasets and benchmarks, establishing new standards for legal NLP research.
Limitations
- Despite employing multiple adaptation strategies, the models still struggle with highly imbalanced label distributions and extremely fine-grained labels, limiting transfer effectiveness in some cases.
- The experiments are primarily confined to European Union legal texts, raising questions about generalizability to other legal systems and languages outside the EU context.
- Computational costs associated with large-scale pretrained models and extensive fine-tuning, especially for resource-constrained environments, remain a challenge. Further research is needed to optimize efficiency and scalability.
- The models show limited ability to generalize to entirely new legal concepts not present in the training data, indicating a need for continual learning or domain adaptation techniques.
Future Work
Future research should explore more sophisticated continual learning frameworks to better handle concept drift over extended periods. Extending the dataset and experiments to non-European legal systems and low-resource languages will enhance model robustness and applicability. Integrating external knowledge bases or legal ontologies could improve understanding of emerging legal concepts. Additionally, developing lightweight models suitable for deployment in real-world legal applications, such as legal search engines or AI assistants, remains a promising direction. Cross-disciplinary efforts combining legal expertise with NLP innovations will be crucial for advancing practical legal AI solutions.
AI Executive Summary
The rapid expansion of multilingual legal information poses significant challenges for automated classification and understanding. Traditional NLP models, often trained on monolingual data, struggle to generalize across languages, especially in domains like law where precision and hierarchical labels are critical. Recognizing this, our research introduces MULTI-EURLEX, a large-scale, multilingual dataset comprising 65,000 EU legal documents translated into 23 languages, annotated with hierarchical EUROVOC labels. This dataset serves as a benchmark to evaluate the capacity of pretrained Transformer models, such as XLM-ROBERTA and MT5, for cross-lingual legal document classification.
Existing models tend to suffer from catastrophic forgetting when fine-tuned on a single language, leading to poor zero-shot transfer performance. To address this, we investigate a suite of adaptation strategies—partial layer freezing, adapters, BITFIT, and LNFIT—that aim to preserve the multilingual knowledge embedded during pretraining. Our experiments reveal that these strategies significantly improve transferability, with adapters consistently outperforming other methods, especially in low-resource and fine-grained label scenarios.
A critical insight from our study is the impact of temporal concept drift. By employing chronological splits that mirror real-world legal evolution, we observe a gradual decline in model performance over time, highlighting the importance of realistic evaluation protocols. In contrast, random splits tend to overestimate the true transfer capabilities, masking the challenges posed by concept drift.
Our comprehensive evaluation across multiple label hierarchies and languages demonstrates that while zero-shot transfer remains challenging, the combination of pretrained models and effective adaptation strategies can substantially narrow the gap. The findings suggest that parameter-efficient methods like BITFIT and LNFIT are promising for scalable deployment, as they require minimal updates yet retain significant multilingual knowledge.
This work advances the state-of-the-art in multilingual legal NLP, providing both a valuable dataset and practical methodologies for robust cross-lingual transfer. It opens avenues for deploying legal AI tools across diverse jurisdictions, reducing language barriers and enhancing access to justice. Future directions include extending the dataset to other legal domains, refining adaptation techniques for better generalization, and integrating external legal knowledge to improve understanding of emerging legal concepts. Ultimately, this research paves the way for more inclusive, efficient, and accurate legal AI systems worldwide.
Deep Dive
Abstract
We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another language (target). We find that fine-tuning a multilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set.
References (20)
Parameter-Efficient Transfer Learning for NLP
N. Houlsby, A. Giurgiu, Stanislaw Jastrzebski et al.
We Need To Talk About Random Splits
Anders Søgaard, S. Ebert, Jasmijn Bastings et al.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Linting Xue, Noah Constant, Adam Roberts et al.
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
Haoxiang Zhong, Chaojun Xiao, Cunchao Tu et al.
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Jonathan Frankle, D. Schwab, Ari S. Morcos
Large-Scale Multi-Label Text Classification on EU Legislation
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis et al.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe, Christian Szegedy
Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective
Nikolaos Aletras, D. Tsarapatsanis, Daniel Preotiuc-Pietro et al.
A survey of cross-lingual embedding models
Sebastian Ruder
Deep Residual Learning for Image Recognition
Kaiming He, X. Zhang, Shaoqing Ren et al.
Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts
Truong-Son Nguyen, Le-Minh Nguyen, S. Tojo et al.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Mikel Artetxe, Holger Schwenk
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau, Guillaume Lample, Ruty Rinott et al.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Alex Wang, Amanpreet Singh, Julian Michael et al.
Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing
Amir Rosenfeld, John K. Tsotsos
Datasheets for datasets
Timnit Gebru, Jamie H. Morgenstern, Briana Vecchione et al.
Explainable Prediction of Medical Codes from Clinical Text
J. Mullenbach, Sarah Wiegreffe, J. Duke et al.
Cited By (20)
EXCISE: Query-Side Exclusion for Late-Interaction Retrieval
A Review of Question-Answering Systems Using Deep Learning in the Arabic Language
Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts
Mimir: Large-scale Multilingual Concept Modeling
LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval
Decoding legal processes: AI-driven system to streamline processing of the criminal records in Moroccan courts
MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP
Enhancing Indian legal judgment classification with embeddings, feature selection, and ensemble strategies
LegalBench.PT: A Benchmark for Portuguese Law
Topic Classification of Interviews on Emergency Remote Teaching
LegalViz: Legal Text Visualization by Text To Diagram Generation
Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods
Continually Adding New Languages to Multilingual Language Models
MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
Scale-Free Characteristics of Multilingual Legal Texts and the Limitations of LLMs
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains
Survey on legal information extraction: current status and open challenges
The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models
Document Intelligence in the Era of Large Language Models: A Survey
Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation