LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models
LookME introduces hierarchical two-level retrieval and sparse injection to enhance multimodal embeddings in vision-language models, outperforming text-only PLE methods with significant efficiency gains.
Key Findings
Methodology
The proposed framework employs a hierarchical two-level retrieval strategy, combining coarse scene-level routing with fine-grained primitive-level search. External multimodal embeddings are stored in ROM, enabling partitioned storage and on-demand access. The scene-level retrieval uses similarity matching with scene embeddings, refined by Gumbel-Softmax, while primitive-level retrieval employs an MLP and cosine similarity to select relevant primitives. Sparse activation (SKA) dynamically gates retrieval pathways at token positions based on attention contributions, reducing overhead. Cross-layer sparse propagation transmits enhancement information efficiently, avoiding repeated retrieval. The entire system integrates with the decoder, supporting multimodal pathways for text, image, and cross-modal embeddings, optimized through sparse intra- and inter-layer activation. The training jointly optimizes embedding tables and model parameters, enabling effective offline retrieval during inference.
Key Results
- On 15 benchmarks, LookME achieves state-of-the-art performance, with improvements of up to 3.9 points over baseline models like Qwen25-VL, especially excelling in fine-grained alignment tasks such as CV-Bench-3D (79.2 vs. 75.3). The model demonstrates a 2.8-point average gain across diverse visual understanding tasks, validating the effectiveness of hierarchical retrieval and sparse injection. Ablation studies confirm that the two-level retrieval reduces retrieval latency while maintaining high accuracy, and cross-layer propagation further enhances efficiency.
- Compared to text-only PLE variants, LookME's multimodal embedding enhancement leads to better evidence aggregation and multi-image alignment, significantly boosting performance in visual question answering, document understanding, and hallucination detection. The approach reduces model parameters and computational costs while achieving superior results, confirming its scalability and robustness.
Significance
This work addresses critical bottlenecks in deploying large-scale multimodal models in resource-constrained environments. By enabling efficient external multimodal knowledge retrieval and integration, it paves the way for more scalable, flexible, and high-performing vision-language systems. The hierarchical retrieval mechanism and sparse injection strategy provide a new paradigm for multimodal information fusion, with broad implications for AI applications such as autonomous vehicles, robotics, and intelligent assistants. The approach also offers a practical solution to the memory and latency issues faced by current dense and MoE models, facilitating real-world deployment.
Technical Contribution
The paper's key technical innovations include the hierarchical two-level retrieval framework tailored for continuous multimodal embeddings, the design of sparse intra- and inter-layer activation modules, and the integration of ROM-based storage with GPU-based fast retrieval. This enables decoupled storage and on-demand access, overcoming ID-based retrieval limitations. The method supports multi-scale, cross-modal, and primitive-level retrieval, with adaptive gating mechanisms that dynamically prioritize critical information, reducing overhead. These advancements collectively expand the capabilities of PLE-style architectures to multimodal domains, offering a scalable and efficient solution for large-scale vision-language models.
Novelty
This is the first work to extend PLE-style external embedding retrieval to continuous, multi-modal embeddings via a hierarchical two-level approach. It introduces a novel sparse injection mechanism that adaptively activates retrieval pathways based on token importance, and supports cross-layer sharing of embedding tables. Unlike prior methods limited to discrete text tokens, this framework handles rich visual and cross-modal features, enabling scalable, resource-efficient multimodal knowledge integration with minimal overhead.
Limitations
- Despite its efficiency, the hierarchical retrieval may still face challenges with extremely high-dimensional or highly dynamic multimodal data, potentially impacting latency and accuracy.
- The reliance on sparse activation and cross-layer propagation assumes certain attention patterns, which may not hold in all scenarios, possibly leading to information loss.
- Current validation focuses on visual and cross-modal tasks; extending to audio, video, or other modalities requires further adaptation and testing.
Future Work
Future research will explore adaptive layer-level routing strategies, more sophisticated sparse gating mechanisms, and broader modality support. Enhancing the robustness of hierarchical retrieval in dynamic environments and integrating self-supervised learning for better embedding alignment are also promising directions. Additionally, efforts will focus on real-time deployment in edge devices and expanding the framework's applicability to multi-task, multi-modal scenarios.
AI Executive Summary
The rapid growth of vision-language models (VLMs) has brought remarkable capabilities in multimodal understanding, yet scalability remains a significant challenge. As models expand to hundreds of billions of parameters, their deployment in resource-constrained environments becomes increasingly difficult due to high memory and computational demands. Traditional dense models and Mixture-of-Experts (MoE) architectures, while powerful, suffer from latency and storage bottlenecks, especially when large expert weights need frequent loading. Recent innovations like Google DeepMind’s Per-Layer Embedding (PLE) architecture introduced external embedding tables stored in ROM, enabling lightweight lookups to enhance token representations without burdening GPU memory. However, existing PLE approaches primarily focus on discrete text embeddings, limiting their effectiveness in multimodal settings where visual and cross-modal embeddings are continuous and richer in information.
This paper introduces LookME, a novel framework that extends PLE-style external embedding retrieval to multimodal embeddings. The core idea is to employ a hierarchical two-level retrieval strategy, first coarse-routing scene-level embeddings, then fine-grained primitive-level features, supported by a soft selection mechanism. This approach addresses the challenge of efficiently retrieving continuous multimodal embeddings from large-scale tables without online joint loading, leveraging a decoupled storage and retrieval paradigm. To further improve efficiency, the authors incorporate a sparse injection strategy, dynamically prioritizing critical embeddings within layers and enabling cross-layer reuse, significantly reducing overhead.
Experimental results across multiple visual benchmarks demonstrate that LookME outperforms existing text-only PLE methods and baseline vision-language models, achieving notable improvements in tasks like visual question answering, 3D understanding, and hallucination detection. The model’s ability to balance performance, efficiency, and resource utilization marks a significant step forward in deploying large-scale multimodal models in real-world applications. Despite some limitations in handling extremely high-dimensional data, the proposed hierarchical retrieval and sparse injection provide a flexible, scalable solution for future multimodal AI systems, with promising potential in autonomous driving, robotics, and intelligent assistants.
Deep Dive
Abstract
Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding. However, scaling dense or sparse Mixture-of-Experts (MoE) models to improve performance limits deployment in resource-constrained environments due to the trade-off between high memory usage from full loading and increased latency from on-demand loading. Recently, the Per-Layer Embedding (PLE) architecture addresses this by scaling models with large external embedding tables stored in read-only memory (ROM) and performing lightweight lookup to retrieve relevant embeddings to enhance token representations. Nevertheless, existing PLE-style methods are primarily designed for text embeddings due to the convenience of ID-based retrieval, limiting their effectiveness in VLMs where multimodal embeddings contain richer information for visual tasks. In this paper, we propose LookME, the first framework that enables lookup-based enhancement for multimodal embeddings in VLMs while supporting partitioned storage and on-demand loading. To efficiently lookup arbitrary continuous multimodal embeddings from large-scale embedding tables, we propose a hierarchical two-level lookup method employing a coarse-to-fine strategy that performs lookups from the scene-level to the intra-scene primitive-level. Furthermore, we integrate the lookup method with a sparse injection strategy, which adaptively prioritizes critical embeddings over voluminous multimodal embeddings within layers, and facilitates embedding table reuse across neighboring layers, improving the trade-off among efficiency, model size, and performance. Experiments on multiple visual benchmarks show that LookME outperforms text-only PLE-style methods, validating the effectiveness of lookup-based multimodal embedding enhancement.