Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods

The evolution of large language models (LLMs) like GPT and BERT has underscored the importance of vast, high-quality datasets. Early efforts relied on web crawling and heuristic filtering, but issues such as data noise, bias, and lack of diversity persisted. Recent research has explored data reweighting techniques—such as gradient influence methods (e.g., LESS) and semantic similarity-based weighting—to improve generalization and training efficiency. However, these methods often operate offline, fixing data importance before training begins, which limits their adaptability to model evolution and task shifts. The challenge remains to develop a principled, scalable approach that dynamically adjusts data contribution during training, preserving diversity while focusing learning on relevant samples.

Existing data curation strategies are predominantly static, involving pre-processing steps that select or filter data before training. This static nature ignores the evolving needs of the model as it learns, leading to potential overfitting on certain data subsets and underutilization of valuable information. Hard filtering reduces data diversity, impairing the model’s ability to generalize across tasks and domains. Moreover, re-running the entire pipeline under data or task shifts is costly and impractical. The core problem is how to design a data importance mechanism that adapts in real time, reflecting the model’s current state and the relevance of samples, to improve robustness and efficiency.

ADAPT introduces a unified online reweighting framework that leverages similarity signals—such as cosine similarity of dense embeddings and lexical overlap metrics—to evaluate sample relevance during training. It employs an adaptive per-sample learning rate, scaled by these signals, allowing the model to focus more on informative samples as training progresses. This approach unifies data selection, mixing, and reweighting under a single theoretical formalism, enabling real-time, model-aware adjustment of sample contributions. The method maintains the full dataset, avoiding the pitfalls of hard filtering, and forms an implicit curriculum that shifts focus from broad patterns to semantic nuances, significantly enhancing generalization and robustness.

• �� Collect similarity signals between training samples and validation set using metrics like cosine similarity of embeddings and lexical overlap (BM25). • Compute dense representations of samples via model’s last-layer hidden states, applying weighted pooling to emphasize later tokens. • Normalize similarity scores through sigmoid functions to produce absolute weights, ensuring stability and robustness. • Update sample weights dynamically during training, based on current model state and similarity signals, using a lightweight computation process. • Incorporate these weights into gradient updates by scaling per-sample gradients, effectively adjusting the learning rate per sample. • Refresh embeddings periodically to reflect evolving model representations, maintaining alignment with current training dynamics. • Ensure computational efficiency by limiting embedding updates and similarity calculations, suitable for large-scale training.

The framework was evaluated on instruction tuning with datasets like Flan V2, Dolly, and Open Assistant, and on large-scale pretraining with SlimPajama corpus using Llama-2-7B. Metrics included accuracy on MMLU and out-of-domain benchmarks like BBH. Baselines included offline methods (LESS, RegMix) and online strategies (LinUpper). Experiments involved training with 50B tokens under a fixed compute budget, comparing performance, robustness, and generalization. Hyperparameters such as similarity thresholds, refresh intervals, and embedding update frequency were tuned through ablation studies. Results consistently showed that ADAPT outperformed baselines, with performance gains up to 7.2% in accuracy per FLOP and improved out-of-domain transfer.

ADAPT achieved a 7.2% average accuracy increase per FLOP over offline methods like LESS, with particularly notable improvements in out-of-domain benchmarks (BBH, RACE). It demonstrated superior robustness across tasks, maintaining data diversity and avoiding overfitting. The adaptive weighting led to better semantic alignment, reflected in higher zero-shot and few-shot performance. Ablation studies confirmed the importance of similarity signals and embedding update frequency, validating the design choices. Overall, the results establish ADAPT as a scalable, effective method for enhancing model generalization and training efficiency.

The method is applicable in large-scale pretraining, instruction tuning, and domain adaptation scenarios where data quality varies. It benefits organizations aiming to reduce data curation overhead, improve robustness, and adapt models to shifting data distributions. The approach can be integrated into existing training pipelines with minimal overhead, enabling more flexible, data-efficient model development, especially in dynamic environments like real-time data streams or multi-task learning.

The reliance on similarity signals makes the method sensitive to noisy or biased data representations, potentially affecting stability. Embedding computation, while lightweight, still incurs overhead at scale, especially with very large models or frequent updates. Its effectiveness in highly noisy or adversarial datasets remains to be validated. Future work should focus on robustness enhancements, multi-modal signals, and theoretical guarantees for convergence and stability.