To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios
This study compares MRL-trained and non-MRL models under various truncation levels, showing non-MRL models outperform MRL ones below 80% truncation.
Key Findings
Methodology
The paper conducts comparative experiments on models trained with and without MRL, truncating embeddings at 10% to 80% levels. Using datasets like NanoBEIR and MTEB, it evaluates downstream tasks such as retrieval and classification. The analysis includes performance metrics (e.g., nDCG@10, accuracy) and internal feature variance to understand the mechanisms behind robustness. Multiple models, including BERT, RoBERTa, Qwen3, and EmbeddingGemma, are tested to ensure broad applicability. The experiments also analyze the variance of low-dimensional features to reveal how information storage differs between MRL and non-MRL models.
Key Results
- Results show that unless truncation exceeds 80%, non-MRL models outperform MRL models in most tasks, with performance drops less than 5%. In extreme truncation (>80%), MRL models demonstrate better robustness, confirming their advantage in high compression scenarios. Feature variance analysis indicates MRL increases low-dimensional feature variance, but this does not necessarily translate into overall performance gains. The findings suggest that inherent learned representations contribute more to robustness than training strategies alone.
- Across multiple models and datasets, random truncation often yields better performance than truncation on MRL-trained models, especially at moderate compression levels. This challenges the assumption that MRL training always enhances robustness and highlights the efficiency of simple random truncation for cost-effective model compression.
- The study also finds that internal feature variance in MRL models is higher in lower dimensions, indicating more information stored there. However, this does not universally improve performance, especially outside high truncation scenarios. Overall, the results advocate for reconsidering the necessity of MRL in practical compression tasks.
Significance
This work questions the widespread reliance on MRL for robustness, showing that simple random truncation often suffices, reducing training complexity and costs. It impacts both academia and industry by providing a cost-effective approach to model compression without sacrificing performance in typical use cases. The insights into feature variance and information storage mechanisms deepen understanding of embedding robustness, guiding future model design and training strategies. This could accelerate deployment of NLP models on resource-constrained devices, broadening their accessibility and application scope.
Technical Contribution
The paper provides a systematic comparison between MRL and random truncation, demonstrating that in most cases, random truncation performs equally or better. It introduces a feature variance analysis to explain internal information distribution, revealing that MRL increases low-dimensional variance but does not necessarily improve overall robustness. The experimental framework spans multiple models, datasets, and tasks, establishing a comprehensive benchmark for future research. The findings suggest that training strategies focusing on inherent robustness may be more effective than complex multi-scale objectives, opening new avenues for efficient model design.
Novelty
This is the first comprehensive study directly comparing MRL with simple random truncation across diverse models and tasks. It challenges the prevailing assumption that MRL inherently enhances robustness, showing that in most practical scenarios, random truncation suffices. The introduction of feature variance analysis as a diagnostic tool offers new insights into the internal mechanisms of embedding robustness, representing a significant departure from prior work that focused solely on performance metrics.
Limitations
- The experiments are limited to models with fewer than 5 billion parameters; larger models may behave differently, requiring further validation. The training data and strategies used are conventional, and more advanced training regimes could alter results. The feature variance analysis focuses mainly on low-dimensional features, leaving high-dimensional dynamics less understood. The study emphasizes moderate to high truncation levels, so the performance at lower truncation ratios remains less explored. Future work should include larger models, diverse training objectives, and more detailed internal analyses.
Future Work
Future research should explore hybrid approaches combining MRL and random truncation, aiming to optimize performance across all truncation levels. Extending experiments to larger models and multi-modal data will test the generality of findings. Investigating different training objectives and regularization techniques could further enhance intrinsic robustness. Theoretical work on information distribution within embeddings may lead to new compression algorithms. Additionally, developing adaptive truncation strategies based on internal feature analysis could improve practical deployment in resource-constrained environments.
AI Executive Summary
Text embeddings are fundamental in NLP applications like retrieval, classification, and recommendation. As models grow larger, compressing these embeddings without losing performance becomes critical. Matryoshka Representation Learning (MRL) was introduced to enable multi-scale representations, allowing models to provide effective embeddings at various sizes by training with multiple truncation objectives. This approach has been widely adopted, with the assumption that it enhances robustness to truncation. However, recent empirical evidence suggests that simple random truncation may achieve comparable robustness, especially at moderate compression levels.
This study systematically compares MRL-trained models with non-MRL models across multiple architectures, including BERT, RoBERTa, and self-trained models like Qwen3 and EmbeddingGemma. Using datasets like NanoBEIR and MTEB, the authors evaluate performance at different truncation ratios, from 10% to over 80%. Results reveal that unless embeddings are heavily truncated (>80%), non-MRL models often outperform MRL models, challenging the assumption that MRL is necessary for robustness. The performance gap narrows at extreme truncation, where MRL models show advantages.
Further analysis of internal feature variance indicates that MRL increases the variance of low-dimensional features, suggesting more information storage in these dimensions. Nonetheless, this does not necessarily translate into better downstream performance. The findings imply that the robustness of embeddings may be an inherent property of the learned representations rather than solely a result of training strategies like MRL.
Overall, the research advocates for reconsidering the cost-effectiveness of MRL, highlighting that simple random truncation is often sufficient for practical purposes. It opens new directions for developing efficient, robust embeddings suitable for resource-constrained environments, emphasizing that model internal properties play a key role in robustness. Future work may explore hybrid methods, larger models, and theoretical frameworks to further optimize embedding compression and robustness.
Deep Dive
Abstract
Matryoshka Representation Learning (MRL) is a widely adopted approach for training text encoders so they provide useful text representations at various sizes, available by simply truncating the resulting vectors at sizes pre-determined at training time. Recent works have shown that randomly truncating text embeddings has minimal impact in downstream performance unless vectors are reduced in size by at least 70%, suggesting that embeddings are already robust to truncation without the use of MRL. However, no prior work has compared random truncation to MRL, so it is unclear how the two methods compare as effective embedding reduction methods. In this paper, we study this by applying the same truncation used by MRL to models trained with and without MRL. Our results across several models and downstream tasks show that, unless heavily truncating embeddings (i.e. reducing their size by at least 80%), truncated embeddings of non-MRL models are competitive with, and often outperform models trained with MRL. This suggests that truncation robustness may not necessarily come from MRL, and that the choice of spending the additional training cost of MRL depends on whether heavy truncation is desired. We make our code available for reproduction.