Latent Visual Reasoning
Latent Visual Reasoning (LVR) enables end-to-end reasoning in visual embedding space, boosting perception-intensive visual question answering by 5% (71.67% vs 66.67%).
Key Findings
Methodology
LVR projects images into a shared semantic space via a visual encoder, then trains the language model to reconstruct key visual tokens in a latent state. It employs a two-stage training: supervised fine-tuning (SFT) with reconstruction and next-token prediction, plus reinforcement learning using the GRPO algorithm to refine latent reasoning. This approach allows the model to perform autoregressive reasoning directly in the visual embedding space, leading to significant improvements in perception-heavy tasks, with an accuracy of 71.67% on MMVP, outperforming Qwen2.5-VL's 66.67%.
Key Results
- On the MMVP benchmark, LVR improves accuracy by 5 percentage points over baseline models, demonstrating enhanced fine-grained visual understanding. The model effectively captures visual details and spatial relations, especially in perception-dependent tasks.
- Reinforcement learning with GRPO further enhances robustness, enabling better balancing between visual semantics reconstruction and textual response generation, leading to more coherent and accurate answers.
- Across multiple perception-heavy tasks such as object counting, spatial reasoning, and image reconstruction, LVR consistently outperforms traditional models, showing particular strength in detail-dependent scenarios.
Significance
This work advances multimodal AI by shifting reasoning from static, external visual features to dynamic, joint semantic space operations. It addresses the bottleneck of shallow visual understanding in existing models, paving the way for more human-like visual reasoning capabilities, crucial for applications like autonomous agents and assistive AI systems.
Technical Contribution
The paper introduces a novel architecture that integrates visual reasoning into the latent space of large language models, combining a reconstruction loss and a modified reinforcement learning algorithm (GRPO) for training stability and scalability. Special tokens enable seamless switching between latent reasoning and text generation, enhancing interpretability and performance.
Novelty
This is the first implementation of direct visual reasoning within the latent embedding space of multimodal large language models, moving beyond reliance on external tools or static visual features. It mimics human visual thinking by reconstructing semantics internally, offering a fundamentally new approach to perception and reasoning integration.
Limitations
- The model's reasoning capacity in extremely complex or cluttered visual scenes remains limited by the expressiveness of the latent space. Further improvements are needed to handle high-density visual content.
- Training costs are high, especially during reinforcement learning, which restricts scalability and real-world deployment.
- Dependence on ROI annotations during supervised training may limit generalization; future work should explore weakly supervised or unsupervised approaches.
Future Work
Future research will focus on reducing reliance on annotated regions, enhancing the model's ability to infer visual semantics autonomously, and extending LVR to more diverse and complex visual tasks. Additionally, integrating multi-task learning and exploring more efficient training algorithms will be key to broader adoption.
AI Executive Summary
Multimodal large language models (MLLMs) have revolutionized the way AI systems understand and generate across visual and textual modalities. However, traditional approaches treat visual information as static inputs, limiting the depth of visual reasoning. Existing methods, such as external tool-based editing or reasoning solely in text space, often fail to capture fine-grained visual details essential for perception-heavy tasks.
This paper introduces Latent Visual Reasoning (LVR), a paradigm shift that enables autoregressive reasoning directly within the visual embedding space. By projecting images into a shared semantic space and training the language model to reconstruct key visual tokens in a latent state, LVR mimics human visual thinking more closely. The architecture builds upon the Qwen-2.5-VL series, incorporating a visual encoder, a multimodal projector, and a modified LLM with special tokens to switch between reasoning modes.
The training process involves two stages: supervised fine-tuning with reconstruction and next-token prediction, and reinforcement learning using a modified GRPO algorithm that rewards accurate and format-compliant responses. This combination ensures the model learns to perform deep visual reasoning while maintaining language generation quality.
Experimental results on perception-intensive benchmarks such as MMVP show that LVR achieves 71.67% accuracy, surpassing previous models like Qwen2.5-VL by over 5 percentage points. The model demonstrates superior performance in tasks requiring detailed visual understanding, spatial reasoning, and object recognition, validating the effectiveness of latent reasoning.
Overall, LVR represents a significant advancement in multimodal AI, bridging the gap between visual perception and reasoning. It offers a scalable, interpretable, and human-like approach to visual understanding, with promising implications for future AI systems capable of more natural and accurate perception and reasoning. Future work will aim to reduce annotation dependence, improve reasoning in complex scenes, and expand application domains, fostering more intelligent and perceptive AI agents.
Deep Dive
Abstract
Multimodal Large Language Models (MLLMs) have achieved notable gains in various tasks by incorporating Chain-of-Thought (CoT) reasoning in language spaces. Recent work extends this direction by leveraging external tools for visual editing, thereby enhancing the visual signal along the reasoning trajectories. Nevertheless, these approaches remain fundamentally constrained: reasoning is still confined to the language space, with visual information treated as static preconditions. We introduce Latent Visual Reasoning (LVR), a new paradigm that enables autoregressive reasoning directly in the visual embedding space. A visual encoder first projects images into visual tokens within a joint semantic space shared with the language model. The language model is then trained to generate latent states that reconstruct key visual tokens critical for answering the query, constituting the process of latent visual reasoning. By interleaving LVR with standard text generation, our model achieves substantial gains on perception-intensive visual question answering tasks. In addition, we adapt the GRPO algorithm to conduct reinforcement learning on latent reasoning, further balancing LVR and textual generation. We show that LVR substantially improves fine-grained visual understanding and perception, achieving 71.67% on MMVP compared to 66.67% with Qwen2.5-VL. Code base and model weights will be released later.