Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
Study enhances legal QA performance via context-injected fine-tuning, showing significant gains for Qwen3.5 at 0.8B parameters.
Key Findings
Methodology
The study employs context-injected fine-tuning by embedding relevant legal text into QA pairs during training to enhance context utilization at inference. Experiments were conducted using Qwen3.5 at 0.8B, 2B, and 4B parameters.
Key Results
- At 0.8B parameters, fine-tuning improved the 2022 English FAISS score from 2 to 34, demonstrating significant performance enhancement.
- The 2B model also showed improvements in most conditions, although the 4B model regressed in English scenarios.
- Fine-tuning drastically reduced language drift from Bangla to English, from 53.2% to 0.7%.
Significance
This research demonstrates that context-injected fine-tuning can significantly enhance the performance of small language models in legal QA, especially in low-resource languages like Bangla, providing new avenues for accurate legal information retrieval.
Technical Contribution
Introduces a novel fine-tuning method that improves context utilization by embedding legal text during training. Achieves higher performance gains on small models compared to existing methods.
Novelty
First application of context-injected fine-tuning in Bangladeshi legal QA, showing significant performance improvements, particularly in low-resource language settings.
Limitations
- The 4B model did not show significant improvement in English conditions, possibly due to model complexity.
- Inconsistent performance across different language conditions, especially between English and Bangla.
Future Work
Future work could explore larger datasets and more complex model architectures to further enhance performance in multilingual legal QA.
AI Executive Summary
Access to legal aid in Bangladesh remains a challenge for many. Existing legal QA models perform poorly when handling Bangla, especially smaller models. This paper proposes a novel context-injected fine-tuning method that embeds relevant legal text during training to enhance context utilization at inference.
The study uses the Qwen3.5 model, conducting experiments at 0.8B, 2B, and 4B parameters. Results show significant performance improvements for the 0.8B model post-fine-tuning, particularly under the 2022 English FAISS retrieval condition, where scores jumped from 2 to 34. Additionally, fine-tuning significantly reduced language drift from Bangla to English.
Although the 4B model did not show significant improvement in English conditions, the findings suggest that context-injected fine-tuning holds potential for enhancing small legal QA models. Future work could explore larger datasets and more complex model architectures to further improve performance in multilingual legal QA.
Deep Analysis
Background
Legal QA systems face challenges in Bangladesh, particularly in low-resource language settings. Existing models perform poorly with Bangla, affecting the accuracy of legal information retrieval.
Core Problem
Small language models often answer incorrectly even with legal text provided, especially in low-resource languages like Bangla. Improving context utilization is a key challenge.
Innovation
The proposed context-injected fine-tuning method embeds legal text during training to enhance context utilization at inference, achieving significant performance gains on small models.
Methodology
- �� Experiments conducted using Qwen3.5 at 0.8B, 2B, and 4B parameters.
- �� Relevant legal text embedded into QA pairs during training.
- �� Evaluation using 2022 and 2023 Bangladesh Bar Council exam questions.
Experiments
Experiments used 2,165 bilingual QA records curated from six Bangladeshi acts and three schedules. Evaluation employed 2022 and 2023 Bangladesh Bar Council exam questions.
Results
At 0.8B parameters, fine-tuning improved the 2022 English FAISS score from 2 to 34. Fine-tuning also drastically reduced language drift from Bangla to English.
Applications
The method can enhance legal QA systems' performance in low-resource language settings, aiding more people in accessing accurate legal information.
Limitations & Outlook
The 4B model did not show significant improvement in English conditions, possibly due to model complexity. Future work could explore larger datasets and more complex model architectures.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe (legal text), but sometimes you forget the details. Context-injected fine-tuning is like having a sticky note next to your recipe with key steps. Even if you forget some details, you can check the note to ensure you're doing it right. This method helps the model better utilize legal text, just like the note helps you cook better.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to answer questions. You have a hint card with important info. Context-injected fine-tuning is like using this hint card in the game to help you answer questions better. Even if the questions are tough, you can find the right answers using the hint card. This method makes the model smarter at answering legal questions, just like you have an edge in the game!
Glossary
Context Injection
Embedding relevant information into the model during training to enhance context utilization at inference.
Used in legal QA to improve model performance by embedding legal text.
Fine-Tuning
Training a pre-trained model on a specific task to improve its performance on that task.
Fine-tuning Qwen3.5 to enhance legal QA performance.
Qwen3.5
A small language model with varying parameter scales (0.8B, 2B, 4B).
Used in experiments for Bangladeshi legal QA.
FAISS
A library for efficient similarity search, commonly used in information retrieval.
Used in experiments to retrieve relevant legal text.
BM25
A popular information retrieval algorithm used to calculate document-query relevance.
Used in experiments to retrieve relevant legal text.
Open Questions Unanswered questions from this research
- 1 How to validate the effectiveness of context-injected fine-tuning on larger datasets?
- 2 Can this method be applied in other low-resource language settings?
Applications
Immediate Applications
Legal Consultation
Helps legal practitioners quickly access accurate legal information in low-resource language settings.
Long-term Vision
Global Legal QA System
Develop a multilingual legal QA system to help users worldwide access legal information.
Abstract
A small language model can receive the governing statutory provision and still answer incorrectly. We test whether fine-tuning on examples containing relevant law improves later use of retrieved law. We curate 2{,}165 bilingual QA records from six Bangladeshi acts and three schedules, then fine-tune Qwen3.5 at 0.8B, 2B, and 4B. Evaluation uses the 2022 and 2023 Bangladesh Bar Council exams in Bangla and machine-translated English, with no retrieval, BM25, or FAISS, scored by strict consistency over three seeded runs. At 0.8B, fine-tuning raises the 2022 English FAISS score from 2 to 34 of 100. Gains at 0.8B and 2B survive paired testing, but the 4B model has no detectable net gain: Bangla improves while several English conditions regress. Fine-tuning also reduces answers that drift from Bangla into mostly English from 44.0--53.2\% to 0.2--0.7\%, with adjusted $p<.001$ at every scale. Retrieval quality is therefore not the only bottleneck. Small bilingual legal models also differ in how they use supplied law and whether they answer in the requested language. The dataset is publicly available at https://huggingface.co/datasets/momahadi/bangladesh-legal-qa-dataset.