CRAFT: Clustered Regression for Adaptive Filtering of Training data

As the field of natural language processing rapidly evolves, the performance of machine translation models critically depends on the quality and relevance of their training data. In recent years, parallel corpora have expanded to tens of millions of sentence pairs, such as the 33 million English-Hindi pairs in the NLLB corpus. However, full fine-tuning on such large-scale corpora is computationally expensive and often unnecessary. Selecting a small, well-chosen subset can match or exceed the performance of training on the full dataset, provided the selection captures the right distributional properties. Existing methods address this problem with varying trade-offs between quality and computational cost, such as lexical methods that are fast but fail to capture semantic structure, and gradient-based methods that achieve strong performance but require expensive encoder inference or optimal transport solves over the full candidate pool.

Selecting appropriate training data from large-scale corpora has emerged as a critical challenge. Full fine-tuning is not only computationally expensive but often unnecessary. A small, well-chosen subset can match or exceed the performance of training on the full dataset, provided the selection captures the right distributional properties. Existing methods address this problem with varying trade-offs between quality and computational cost, such as lexical methods that are fast but fail to capture semantic structure, and gradient-based methods that achieve strong performance but require expensive encoder inference or optimal transport solves over the full candidate pool.

The CRAFT method decomposes the source-target joint distribution and employs a two-stage selection strategy: (i) match the validation source distribution through proportional budget allocation across k-means clusters, and (ii) within each source cluster, select training pairs whose target embeddings minimize a conditional expected distance derived from the validation target distribution. This method proves that proportional cluster allocation bounds the continuous KL divergence between selected and validation distributions, with the residual controlled by cluster diameters. CRAFT fundamentally differs from existing state-of-the-art methods by independently clustering source and target embeddings to capture conditional structures in the validation set, avoiding joint embedding processing of traditional methods.

• �� CRAFT decomposes the source-target joint distribution and employs a two-stage selection strategy.
• �� First, it matches the validation source distribution through proportional budget allocation across k-means clusters.
• �� Then, within each source cluster, it selects training pairs whose target embeddings minimize a conditional expected distance derived from the validation target distribution.
• �� This method proves that proportional cluster allocation bounds the continuous KL divergence between selected and validation distributions, with the residual controlled by cluster diameters.
• �� CRAFT fundamentally differs from existing state-of-the-art methods by independently clustering source and target embeddings to capture conditional structures in the validation set, avoiding joint embedding processing of traditional methods.

In experiments, CRAFT excels in English-Hindi translation tasks. Using 33 million NLLB sentence pairs to fine-tune the mBART model, CRAFT achieves a BLEU score of 43.34, outperforming TSDS by 2.13 points. Additionally, CRAFT completes selection 40 times faster, with TF-IDF vectorization allowing the entire process to complete in under one minute on CPU, whereas TAROT takes 75.6 seconds, and CRAFT only takes 26.86 seconds, a 2.8x speedup. The CRAFT method significantly reduces computational costs while improving model performance by selecting high-quality subsets for fine-tuning from large-scale corpora.

CRAFT excels in English-Hindi translation tasks. Using 33 million NLLB sentence pairs to fine-tune the mBART model, CRAFT achieves a BLEU score of 43.34, outperforming TSDS by 2.13 points. Additionally, CRAFT completes selection 40 times faster, with TF-IDF vectorization allowing the entire process to complete in under one minute on CPU, whereas TAROT takes 75.6 seconds, and CRAFT only takes 26.86 seconds, a 2.8x speedup.

The CRAFT method significantly reduces computational costs while improving model performance by selecting high-quality subsets for fine-tuning from large-scale corpora. It addresses the shortcomings of traditional methods in handling source-target conditional relationships, especially in multilingual translation tasks.

CRAFT may be affected by clustering quality when dealing with very high-dimensional embeddings, leading to a mismatch between the selected subset and the validation distribution. Additionally, in some low-resource language pairs, the conditional relationship between source and target may not be apparent, affecting CRAFT's selection effectiveness. Future research can explore the application of CRAFT in other tasks, such as image classification or text generation.