GptGet

BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translation

TL;DR

BhashaSetu improves low-resource translation quality via corpus deduplication, reducing performance by 1.17 BLEU and 2.21 chrF++.

cs.CL 🟑 Intermediate 2026-05-26 5 views
Param Thakkar Anushka Yadav Michael Tiemann Abhi Mehta Akshita Bhasin Shrinivas Khedkar
AI Reader Arxiv Page Download PDF
machine translation low-resource languages dataset corpus deduplication model fine-tuning

Key Findings

Methodology

BhashaSetu employs a data-centric approach, enhancing English-Marathi translation quality through corpus deduplication and morphological processing. Parameter-efficient fine-tuning of NLLB-200-distilled-600M using LoRA validates the significant impact of deduplication on translation quality.

Key Results

  • Deduplication impact: Removing deduplication reduces performance by 1.17 BLEU and 2.21 chrF++, highlighting its importance in translation quality improvement.
  • Experiments show BhashaSetu dataset excels in multi-domain translation tasks, especially in news and politics.
  • Comparative experiments confirm stable improvements in translation quality through morphological processing.

Significance

This research addresses the persistent data scarcity in low-resource language translation by creating a multi-domain English-Marathi parallel corpus. Through deduplication and morphological processing, it significantly improves translation quality, offering new directions for low-resource language machine translation research.

Technical Contribution

Technical contributions include developing a generalized language-aware preprocessing framework suitable for morphologically rich languages. Corpus deduplication and morphological processing enhance model performance, validating the importance of data quality.

Novelty

BhashaSetu is the first dataset to significantly improve low-resource language translation quality through corpus deduplication, providing a new perspective to tackle translation challenges in morphologically complex languages.

Limitations

  • The dataset's formal text bias may lead to suboptimal performance on informal or dialectal inputs.
  • High proportion of religious texts may introduce topical and lexical bias.
  • Future iterations should include conversational and dialectal data to enhance model adaptability.

Future Work

Future work includes expanding the dataset to cover more conversational and dialectal data, developing more adaptive translation models, and exploring translation challenges in other low-resource languages.

AI Executive Summary

The BhashaSetu project aims to tackle the problem of data scarcity in low-resource language translation. Existing English-Marathi parallel corpora are often domain-specific, failing to meet diverse translation needs. BhashaSetu improves translation quality by creating a corpus spanning news, politics, healthcare, literature, and culture.

Using a data-centric approach, BhashaSetu enhances English-Marathi translation quality through corpus deduplication and morphological processing. Experiments show deduplication is crucial, with performance dropping by 1.17 BLEU and 2.21 chrF++ when removed.

The release of BhashaSetu offers new directions for low-resource language machine translation research. By providing a high-quality multi-domain parallel corpus, it promotes reproducible, linguistically informed low-resource machine translation research.

Deep Dive

Abstract

We present BhashaSetu, a linguistically enriched English--Marathi parallel dataset addressing persistent data limitations in low-resource neural machine translation (NMT). Marathi, spoken by over 95 million people, remains underrepresented in high-quality parallel corpora across diverse domains. Our dataset comprises 2.78 million sentence pairs from heterogeneous sources including news, politics, healthcare, literature, and culture, with stemmed and lemmatized representations to support morphology-aware analysis. We benchmark multiple state-of-the-art translation models using BLEU, spBLEU, chrF++, and TER metrics, and conduct parameter-efficient fine-tuning of NLLB-200-distilled-600M using LoRA. A key finding from our ablation: corpus-level deduplication is the single largest preprocessing contributor to downstream quality (removing it reduces performance by 1.17 BLEU and 2.21 chrF++), demonstrating that disciplined cross-source corpus hygiene is a low-cost, high-impact intervention for low-resource, morphologically rich languages. The dataset is publicly released to promote reproducible and linguistically informed low-resource NMT research.

cs.CL cs.LG