Kosmos-2: Grounding Multimodal Large Language Models to the World
KOSMOS-2 is a multimodal large language model with grounding and referring capabilities, achieving state-of-the-art performance on spatial understanding and vision-language tasks.
Key Findings
Methodology
KOSMOS-2 employs a Transformer-based causal language model trained on next-token prediction, integrating grounded image-text pairs (GRIT) where spatial coordinates of bounding boxes are discretized into location tokens. These tokens are linked to text spans via a markdown-like hyperlinked format, enabling the model to learn joint representations of language and spatial regions. The training combines multimodal corpora, large-scale grounded data, and instruction tuning, with the model incorporating a grounding token to facilitate spatial reasoning. The architecture includes a vision encoder and a multimodal transformer with approximately 1.6 billion parameters, optimized with AdamW, trained over 60k steps on 256 V100 GPUs.
Key Results
- On Flickr30k Entities, R@1 reaches 78.7%, outperforming previous models like GRILL (18.9%) by a large margin, demonstrating robust zero-shot grounding.
- On RefCOCO, RefCOCO+ and RefCOCOg, the model achieves 60.57%, 61.65%, and 60.2% accuracy respectively, surpassing prior zero-shot approaches.
- In captioning and VQA tasks, CIDEr scores reach 60.3 and VQA accuracy hits 72.4%, confirming strong perception-language performance.
Significance
This work advances multimodal AI by integrating spatial grounding into large language models, enabling precise object localization and natural language understanding simultaneously. It addresses the longstanding challenge of combining spatial reasoning with language comprehension, paving the way for embodied AI systems capable of complex interactions in real-world environments. The approach enhances human-AI communication, improves task accuracy, and broadens application scope across robotics, autonomous vehicles, and assistive technologies.
Technical Contribution
Introducing a novel discretization of continuous spatial coordinates into location tokens, coupled with a hyperlinked input format that binds text spans to spatial regions. This enables end-to-end training of a single Transformer model with spatial grounding capabilities. The creation of the GRIT dataset, a large-scale grounded image-text corpus, provides the necessary data foundation. The model’s architecture and training strategy demonstrate how spatial information can be seamlessly integrated into large-scale multimodal models, setting new benchmarks in grounding and referring tasks.
Novelty
This is the first large-scale multimodal model to discretize bounding box coordinates into location tokens and embed them directly into the input sequence, effectively unifying spatial and linguistic information. Unlike previous models relying on separate detectors or proposals, KOSMOS-2 learns spatial grounding end-to-end, enabling more accurate and flexible spatial reasoning within a single model framework.
Limitations
- Despite improvements, the model struggles with dense, occluded, or highly cluttered scenes where spatial boundaries are ambiguous, affecting localization accuracy.
- Training requires massive computational resources and large-scale annotated grounded data, limiting accessibility and scalability.
- Handling complex multi-object interactions and long, intricate referring expressions remains challenging, necessitating further research into multi-region reasoning.
Future Work
Future directions include enhancing multi-object spatial reasoning, reducing computational costs, and expanding grounded datasets. Incorporating reinforcement learning and self-supervised strategies could improve adaptability and efficiency. Extending the model to dynamic scenes and real-time applications, such as robotics and autonomous driving, is also a key goal.
AI Executive Summary
KOSMOS-2 marks a significant leap in multimodal AI, integrating spatial grounding into a large language model to achieve unprecedented understanding of visual regions and natural language. Traditional models often rely on separate detection modules, limiting end-to-end learning and flexibility. In contrast, KOSMOS-2 discretizes continuous spatial coordinates into location tokens, embedding them within a unified Transformer architecture. This innovation allows the model to learn spatial relationships directly from large-scale grounded datasets (GRIT), which combine image-text pairs with spatial annotations. The model’s training involves a next-token prediction objective, augmented with instruction tuning to improve human-aligned responses.
Experimental results demonstrate that KOSMOS-2 outperforms existing models on multiple benchmarks, including R@1 of 78.7% on Flickr30k Entities and over 60% accuracy on RefCOCO series. It also exhibits strong zero-shot capabilities in captioning and visual question answering, with CIDEr scores of 60.3 and VQA accuracy of 72.4%. These achievements validate the effectiveness of discretized spatial encoding and hyperlinked input formats, enabling end-to-end spatial reasoning within a single model.
The broader impact of this work lies in its potential to develop embodied AI systems capable of precise spatial understanding and natural language interaction. Such systems could revolutionize robotics, autonomous vehicles, and assistive technologies by enabling more intuitive human-machine communication and decision-making. Despite these advances, challenges remain in handling complex multi-object scenes, reducing training costs, and improving multi-region reasoning. Future research will focus on addressing these limitations, integrating reinforcement learning, and expanding grounded datasets, ultimately pushing AI toward more human-like spatial cognition and embodied intelligence.
Deep Dive
Abstract
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.