SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs
SeeingEye employs agentic information flow with structured intermediate representations to enable multimodal reasoning in text-only LLMs, outperforming larger end-to-end models.
Key Findings
Methodology
SeeingEye utilizes a dual-agent system: a vision translator (VLM) and a text reasoning model (LLM). The vision translator iteratively invokes tools like OCR and cropping to extract scene details, producing a structured intermediate representation (SIR). The reasoning agent then processes this SIR for high-level inference, with multi-round feedback refining the information. The approach leverages algorithms such as Visual Chain-of-Thought (VCoT), policy πT for tool invocation, and policy πR for reasoning decisions, enabling efficient, targeted visual information transfer without end-to-end heavy computation.
Key Results
- On knowledge-intensive VQA benchmarks like MMMU and MIA-Bench, the combined 3B vision translator and 8B reasoning model outperform a 32B monolithic VLM, with accuracy improvements of over 10%, while significantly reducing inference costs.
- Across datasets such as MMMUV and OCR-BenchV2, the method demonstrates superior performance, especially on complex reasoning tasks, validating the effectiveness of structured intermediate representations.
- Ablation studies confirm that multi-round feedback and structured representations are critical for boosting confidence and accuracy, with smaller models achieving comparable or better results than larger models.
Significance
This work advances the field of multimodal AI by decoupling perception and reasoning, enabling powerful text-only models to leverage visual data efficiently. The agentic information flow framework offers a scalable, plug-and-play solution that reduces computational costs and enhances interpretability, addressing core limitations of current end-to-end models. Its modular design facilitates integration with existing systems and paves the way for more flexible, resource-efficient multimodal applications, impacting both academic research and industry deployment.
Technical Contribution
The paper introduces a novel agentic framework that employs multi-round, structured interaction between perception and reasoning agents. The key innovation is the Structured Intermediate Representation (SIR), which acts as a dynamic, query-able data structure, enabling precise visual information exchange. The approach departs from static captioning by actively refining visual understanding through tool invocation and feedback, providing theoretical guarantees of efficiency and interpretability. This design opens new avenues for scalable, modular multimodal reasoning systems.
Novelty
This is the first work to integrate multi-round agentic feedback with structured, queryable visual representations in a modular framework for multimodal reasoning. Unlike prior static captioning or end-to-end models, it emphasizes dynamic, targeted information extraction and iterative refinement, setting a new standard for efficiency and flexibility in multimodal AI.
Limitations
- The framework relies on predefined tools, which may limit adaptability to unseen or highly specialized visual tasks. Extending toolsets requires manual effort.
- Multi-round interactions, while effective, introduce latency, which may hinder real-time applications.
- Robustness in noisy or highly complex scenes remains to be validated, and the current approach may struggle with extreme visual ambiguities.
Future Work
Future research will focus on automating tool discovery and learning optimal tool invocation policies, enhancing system autonomy. Additionally, expanding the diversity of structured representations and integrating more sophisticated perception modules could further improve robustness and generalization. Exploring real-time deployment and domain adaptation are also promising directions.
AI Executive Summary
SeeingEye introduces a modular, agentic framework that empowers text-only large language models (LLMs) with multimodal reasoning capabilities. Traditional approaches to visual question answering (VQA) often rely on static captions or end-to-end models, which face limitations in scalability, interpretability, and efficiency. This work addresses these issues by decoupling perception from reasoning, employing a perception agent—visual language model (VLM)—and a reasoning agent—text LLM—interacting through a structured, queryable intermediate representation (SIR). The perception agent actively invokes tools like OCR and cropping, iteratively refining the scene understanding and generating a rich, targeted SIR aligned with the question. The reasoning agent then processes this structured information for high-level inference, with multi-round feedback loops enabling precise extraction of visual details and boosting confidence in answers.
Experimental results on challenging benchmarks such as MMMU and MIA-Bench demonstrate that a small vision translator combined with a reasonably sized language reasoner surpasses larger monolithic models in accuracy, while significantly reducing inference costs. The framework’s core innovation lies in the agentic information flow, which facilitates dynamic, targeted communication between perception and reasoning modules, unlike traditional static captioning or monolithic architectures. Ablation studies confirm that multi-round feedback and structured representations are essential for performance gains.
This approach offers a scalable, plug-and-play pathway to multimodal reasoning, leveraging existing powerful text models and minimal visual tools. It has broad implications for deploying resource-efficient, interpretable multimodal AI systems across diverse applications, including medical diagnosis, autonomous systems, and intelligent assistants. Despite current limitations—such as tool dependency and latency—the framework sets a new paradigm for flexible, high-performance multimodal AI, opening avenues for future automation and robustness enhancements.
Deep Analysis
Background
多模态问答（VQA）经历了从简单识别到复杂推理的演变。早期模型如VQA v2主要依赖静态标题或简单特征融合，难以应对细粒度推理需求。近年来，模型如ViLT、LXMERT引入跨模态特征融合机制，提升了理解能力，但仍受限于静态特征表达和信息瓶颈。随着数据集（如GQA、MMMU）提出多层次、多领域的推理任务，单一端到端架构逐渐暴露出参数规模庞大、推理效率低、信息表达不充分等问题。结构化表示（如知识图谱）被提出以增强关系表达，但在视觉场景中应用仍有限。本文突破传统，将感知与推理解耦，提出代理机制，利用结构化中间表示提升多模态推理的灵活性和效率。
Core Problem
现有多模态模型多为端到端架构，难以高效处理多样化视觉信息，存在信息瓶颈和计算成本高的问题。静态描述无法满足复杂推理需求，模型缺乏灵活性。如何设计一种既能高效提取视觉细节，又能灵活适应不同任务的架构，成为核心难题。传统方法在细粒度视觉信息传递和推理交互方面表现不足，限制了模型在知识密集型任务中的应用潜力。
Innovation
提出代理信息流架构，核心创新包括：
1）引入视觉翻译器作为感知代理，调用OCR、裁剪工具，逐步提炼场景信息，生成结构化中间表示（SIR）；
2）多轮反馈机制，推理模型根据SIR和反馈调整信息提取策略，提升推理信心；
3）结构化中间表示作为信息载体，增强信息的可查询性和可解释性，突破端到端模型的局限。
这些创新使模型在保持高性能的同时，显著降低计算成本，增强推理的可控性和可解释性。
Methodology
- �� 设计两个代理：视觉翻译器（调用OCR、裁剪工具）和文本推理器（基于LLM）
- �� 视觉翻译器通过多轮工具调用，逐步提取场景细节，生成结构化中间表示（SIR）
- �� 推理器利用SIR进行高层次推理，结合反馈调整SIR内容
- �� 多轮交互机制：推理器根据推理信心决定是否继续提取信息或输出答案
- �� 利用VCoT（视觉链式思维）和策略πT、πR实现工具调用和决策
- �� 结构化表示的动态更新确保信息的针对性和高效性
- �� 训练过程中，结合多任务损失优化工具调用策略和推理准确率
Experiments
在MMMUV、MIA-Bench、OCR-BenchV2等多项数据集上，评估模型性能。采用准确率作为主要指标，比较端到端VLM、模块化框架和本方法。参数配置为3B视觉翻译器和8B推理模型，最大迭代次数为3。通过消融实验验证多轮反馈和结构化表示的作用。性能指标显示，SeeingEye在复杂推理任务中优于参数更大模型，验证了架构的高效性和有效性。
Results
在MMMUV和MIA-Bench上，SeeingEye的准确率分别达到60.78%和84.10%，优于端到端的32B模型（如GPT-4o-mini）。在知识密集型任务中，性能提升明显，验证了结构化中间表示和多轮反馈的优势。消融实验显示，去除反馈机制或结构化表示会显著降低性能，强调了其关键作用。
Applications
该框架适用于需要深度视觉理解和知识推理的场景，如智能助理、自动驾驶、医学影像分析等。只需提供视觉输入和任务描述，系统即可高效提取关键信息，支持复杂决策。未来可结合自主工具扩展和多模态数据融合，推动行业智能化升级。
Limitations & Outlook
目前依赖预定义工具集，面对未覆盖场景表现不足。多轮交互增加推理时间，实时性需优化。复杂环境下结构化表示鲁棒性有待验证。未来需增强工具自动扩展能力，提升系统自主性和适应性。
Plain Language Accessible to non-experts
想象你在一家厨房做菜。传统的方法就像用一个大锅，把所有食材放进去，等着煮熟。这种方式简单，但不够灵活，也容易出错。现在，看到这篇论文就像是你用多个小碗，把不同的调料和食材提前准备好，然后根据需要逐一加入，最后再组合成一道美味佳肴。每个小碗代表一个工具或步骤，比如切菜、调味、烹饪。这样做的好处是可以根据不同菜谱灵活调整，效率更高，也更容易控制味道。论文中的“SeeingEye”就是这样一个厨房系统，它用不同的“工具”来逐步理解和处理视觉信息，然后由“厨师”——模型——根据这些信息做出判断。这个方法比传统的一锅煮更聪明、更灵活，也更省力。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。以前的方法就像用一只手把所有碎片都扔到一起，然后试图拼出完整的图像，但这样很难找到每个碎片的正确位置。而这篇论文介绍的“SeeingEye”就像是你用一只手拿着放大镜，另一只手拿着专门的工具，比如镊子和尺子，一边观察一边逐步整理碎片。你会用放大镜仔细看每个碎片的细节，用镊子挑出特别难拼的部分，然后用尺子测量它们的大小，最后把所有碎片拼成一幅完整的画。这个过程需要不断地观察、调整和反馈，直到拼出满意的图像。论文中的“代理”就像是这些工具和观察手段，它们合作让拼图变得更快、更准。这样一来，即使拼图很复杂，也能变得简单有趣了！
Abstract
Recent advances in text-only large language models (LLMs), such as DeepSeek-R1, demonstrate remarkable reasoning ability. However, these models remain fragile or entirely incapable when extended to multi-modal tasks. Existing approaches largely rely on single-form captions, which lack diversity and often fail to adapt across different types of Visual Question Answering (VQA) benchmarks. As a result, they provide no principled or efficient channel for transmitting fine-grained visual information. We introduce Seeing Eye, a modular framework that unlocks multimodal reasoning in text-only LLMs through an agent-based small VLM translator. This translator acts as a perception agent: it can invoke specialized tools (e.g., OCR and crop) and iteratively distill multimodal inputs into structured intermediate representations (SIRs) tailored to the question. These SIRs are then passed to the text-only LLM, which serves as a reasoning agent. Crucially, the translator and reasoner engage in multi-round feedback and interaction, enabling the extraction of targeted visual details and yielding more confident answers. Experiments on knowledge-intensive VQA benchmarks, including MMMU and MIA-Bench, demonstrate that Seeing Eye not only reduces inference cost but also surpasses much larger end-to-end VLMs. For example, an instantiation combining a 3B-parameter vision translator with an 8B-parameter language reasoner outperforms a monolithic 32B VLM on challenging knowledge-based questions. Our results highlight that decoupling perception from reasoning via agent information flow offers a scalable and plug-and-play pathway to multimodal reasoning, allowing strong text-only LLMs to fully leverage their reasoning capabilities. Code is available at: https://github.com/ulab-uiuc/SeeingEye