ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models
Proposed ActiView benchmark evaluates active perception in multimodal LLMs via view shifting and zooming, revealing significant performance gaps.
Key Findings
Methodology
This study introduces ActiView, a benchmark combining view zooming and shifting to simulate human active perception. Tasks involve limited initial views, requiring models to actively adjust their perspective to gather critical information. A curated dataset covers environment, object, and event categories, with visual clues annotated. The evaluation measures question accuracy, view operation counts, and strategy effectiveness across 30 models, including commercial and open-source. The framework emphasizes dynamic exploration, pushing models beyond static input understanding.
Key Results
- Models achieved an average accuracy of 66.40%, significantly below human performance at 84.67%. Proprietary models like GPT-4o outperformed open-source counterparts but still lagged notably, especially in complex scenes. Recent open-source models such as Qwen2.5-VL showed promising improvements, approaching commercial model levels. The number of view operations varied widely, indicating differences in exploration strategies. Performance declined sharply in high-difficulty scenarios, highlighting the need for better active perception strategies.
- Across models, view zooming and shifting accuracy averaged around 68.00% and 65.11%, respectively. The number of operations and view choices reflected models' autonomy and strategy quality. Combining multiple views improved understanding but did not close the gap with humans. The results suggest active perception remains a key bottleneck, with current models limited in their ability to adaptively explore environments, especially under constraints.
- Analysis of difficulty levels revealed that models struggle most with fine-grained details and missing information, with accuracy dropping below 50% in hardest settings. These findings underscore the importance of developing models capable of flexible, goal-directed exploration, which is crucial for real-world applications like autonomous navigation and interactive AI systems.
Significance
This work pioneers a systematic evaluation of active perception in multimodal models, addressing a critical gap in AI research. Active perception is fundamental for models to operate intelligently in real-world environments, where information is incomplete and dynamic. By formalizing view shifting and zooming as core capabilities, the benchmark enables quantification and comparison of models’ exploration strategies. The insights gained guide future development towards more autonomous, adaptable AI systems, with broad implications for robotics, autonomous vehicles, and human-AI interaction. The framework sets a new standard for evaluating dynamic perception, fostering innovation in model architecture and training paradigms.
Technical Contribution
The paper introduces a novel evaluation framework integrating view zooming and shifting, with detailed task design and metrics. It combines visual question answering with active exploration, employing multi-view operation metrics and difficulty stratification. The benchmark's design allows for comprehensive analysis of models’ exploration strategies, highlighting strengths and weaknesses. It also provides a dataset with manually curated instances, visual clues, and multi-category annotations, supporting rigorous evaluation. The approach advances the state-of-the-art by emphasizing dynamic, goal-directed perception, moving beyond static static benchmarks.
Novelty
This is the first comprehensive benchmark explicitly targeting active perception in multimodal models, emphasizing view shifting and zooming as core capabilities. Unlike previous static evaluation datasets, ActiView simulates real-world exploration behaviors, requiring models to actively adjust their perceptual fields. Its multi-category, multi-difficulty design offers a nuanced assessment of models’ exploration strategies, setting a new standard for dynamic perception evaluation. This approach bridges the gap between static understanding and real-world autonomous perception, representing a significant innovation in AI benchmarking.
Limitations
- The current benchmark focuses on static images, lacking dynamic video or real-time interaction scenarios, which are crucial for real-world applications.
- Models still show limited autonomy in view selection, often relying on heuristic strategies rather than goal-driven exploration, indicating room for reinforcement learning integration.
- Evaluation mainly relies on question accuracy and operation counts, which might not fully capture the quality of exploration strategies or the reasoning process behind view adjustments.
Future Work
Future directions include extending the benchmark to dynamic video environments and real-time interaction tasks, integrating reinforcement learning for autonomous view planning, and exploring multi-step reasoning strategies. Additionally, developing more sophisticated metrics to evaluate exploration quality and reasoning depth will be prioritized. The goal is to foster models capable of goal-oriented, adaptive perception in complex, real-world scenarios, advancing towards truly autonomous AI systems.
AI Executive Summary
The rapid advancement of multimodal large language models (MLLMs) has opened new horizons in AI, enabling systems to understand and generate complex visual and textual data. However, current evaluation frameworks predominantly focus on static understanding, neglecting a critical human-like ability: active perception. Active perception involves dynamically adjusting one's sensory focus—shifting and zooming—to gather relevant information in complex environments. Recognizing this gap, the authors introduce ActiView, a novel benchmark designed to assess the active perception capabilities of MLLMs.
ActiView simulates real-world scenarios where models are provided with limited initial views and must actively explore their environment through view shifting and zooming operations. The benchmark encompasses a curated dataset of images spanning environment, object, and event categories, annotated with visual clues to facilitate precise evaluation. Models are tasked with answering visual questions that require multi-step reasoning, involving the identification of missing information, fine-grained details, and contextual cues. The evaluation metrics include question accuracy, view operation counts, and strategy effectiveness, providing a comprehensive picture of models’ exploration abilities.
Experimental results across 30 models reveal a significant performance gap compared to human benchmarks, with the best models achieving around 66% accuracy versus 84% for humans. Proprietary models like GPT-4o outperform open-source counterparts but still show notable limitations, especially in complex scenarios demanding nuanced exploration. The findings highlight that current models lack robust active perception strategies, particularly in dynamically adjusting their perceptual fields to handle intricate visual scenes.
This work's significance lies in establishing a standardized, rigorous framework for evaluating a vital aspect of AI intelligence—active perception. It paves the way for future research integrating reinforcement learning, multi-step reasoning, and dynamic environment understanding, ultimately aiming to develop autonomous systems capable of goal-directed exploration. Despite its strengths, the benchmark's focus on static images limits immediate applicability to real-time video and interactive settings, which are promising directions for future work. Overall, ActiView marks a crucial step toward more intelligent, adaptable AI systems capable of perceiving and reasoning in complex, real-world environments.
Deep Analysis
Background
多模态大模型（MLLMs）近年来经历快速发展，代表性工作如OpenAI的GPT-4V、Liu等推动模型在视觉与语言理解方面取得突破。早期研究多关注静态视觉理解（如视觉问答、图像识别），逐步引入多模态融合技术（如FLAVA、LLaVA），实现跨模态信息交互。然而，这些模型多为被动接受信息，缺乏主动探索能力。主动感知作为人类的核心能力，涉及视野调整、信息筛选与推理，尚未在多模态模型中得到系统性评估。现有评测大多局限于静态场景，难以反映模型在复杂环境中的表现。
Core Problem
当前多模态模型在动态环境中的主动感知能力不足，表现为缺乏视图调整策略，难以自主获取关键信息。传统评估指标如问答准确率无法衡量模型的探索策略和信息筛选能力。实际应用中，模型需在有限视野中主动调整视角，补充缺失信息，理解复杂场景，但现有模型多在静态输入下训练和测试，限制了其在自动驾驶、机器人等领域的应用潜力。亟需开发专门的评估框架，以衡量模型在动态、多变环境中的主动探索能力。
Innovation
本研究提出了创新性的ActiView基准，将视图缩放（zooming）与转移（shifting）结合，模拟人类在复杂场景中的主动探索行为。设计了多类别、多难度的视觉问答任务，要求模型在有限视野下主动调整视角，获取关键信息。引入多视图操作指标，量化模型自主性和策略优化能力。该框架突破了传统静态评估的局限，为主动感知能力的量化提供了新工具，推动多模态模型向更自主、更智能的方向发展。
Methodology
- �� 设计多类别、多难度的视觉问答任务，涵盖环境、对象、事件三大类别。• 采用手工筛选实例，结合视觉线索标注，确保任务的复杂性和多样性。• 构建视图缩放（zooming）和转移（shifting）两大操作流程，模拟人类主动探索行为。• 设定有限初始视野，模型需通过视图调整获取关键信息。• 设计多视图操作指标（如操作次数、选择策略）评估模型自主性。• 实验涵盖30个模型，分析不同操作策略的效果差异。
Experiments
采用包含多类别、多难度实例的视觉问答数据集，模型在不同视野限制下进行测试。评估指标包括问答准确率、视图操作次数、操作策略合理性。实验设计包括单一操作（缩放或转移）和混合操作两种场景，模拟真实环境中的动态感知。通过对比不同模型的表现，分析主动感知能力的差异。还引入人类基准，验证模型在复杂场景中的不足。实验还考察模型在不同难度等级下的表现，揭示模型在细粒度信息捕获方面的局限。
Results
模型在主动感知任务中的平均准确率为66.40%，低于人类84.67%。商用模型如GPT-4o表现优异，但仍远未达到人类水平。开源模型逐步逼近商用模型，Qwen2.5-VL在多视图操作中表现出较强潜力。多模型视图操作次数差异显著，反映自主策略的差异。高难度场景（Shift-H）中模型表现明显下降，说明在细粒度信息捕获和缺失信息补充方面仍有较大提升空间。整体结果表明，主动感知能力是未来模型的重要发展方向。
Applications
该基准适用于自动驾驶、机器人导航、智能监控等场景，模型需在有限视野下自主探索环境，获取关键信息。推动多模态模型在复杂环境中的自主决策能力，提升系统的适应性和鲁棒性。未来可结合强化学习，优化模型的探索策略，实现更高效的主动感知。
Limitations & Outlook
目前基准主要集中于静态图像，尚未覆盖动态视频和连续交互场景，未来应扩展多模态动态场景的评估能力。模型在复杂场景中的表现仍受限于感知策略和推理能力，尚未实现真正的自主探索，需结合强化学习等技术优化。评估指标主要依赖问答准确率，未来应结合行为分析和路径优化等多维指标，全面衡量主动感知的效果。
Plain Language Accessible to non-experts
想象你在一个大厨房里做饭。你不能一次性看到所有的食材和工具，只能看到一部分。你需要不断转动厨房的视角，找到缺少的食材，或者放大某个区域看得更清楚。这个过程就像模型在理解图片时，不仅要看一眼，还要主动转动视角、放大细节，才能知道所有的秘密。就像你在厨房里找调料一样，模型也要主动“找”信息，才能做出正确的回答。这种主动探索比被动接受信息更聪明，也更接近人类的思考方式。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，但你不能一次看到全部的拼图，只能看到一部分。你得不断转动拼图板，找到缺少的拼片，或者放大某个区域看得更清楚。这个过程就像模型在看图片时，不只是静静地看一眼，而是主动转动视角、放大细节，去找到所有隐藏的线索。就像你用手去找拼图的缺口一样，模型也要主动“找”信息，才能拼出完整的答案。这个主动探索的能力，让模型变得更聪明、更像人类，也更能应对复杂的场景。
Glossary
Active Perception (主动感知)
指主动调整视野以获取关键信息的能力，涉及视图转移与缩放。
论文中用来描述模型主动探索环境的能力。
Visual Question Answering (视觉问答)
通过视觉输入回答相关问题的任务，评估模型理解和推理能力。
作为基准任务的核心框架。
Perceptual Field (感知视野)
模型当前能够感知的图像区域，受限于视图大小和位置。
模型需通过操作扩大或转移感知视野。
Zooming (缩放)
放大感知视野中的局部区域以获取细节信息的操作。
主动感知中的关键操作之一。
Shifting (转移)
移动感知视野以补充缺失信息的操作。
模拟人类主动探索行为。
Open Questions Unanswered questions from this research
- 1 如何让模型在复杂动态环境中自主选择最佳视角，提升连续交互中的主动感知能力仍未解决。
- 2 现有模型在多模态融合和细粒度推理方面的能力有限，需结合强化学习等技术进行突破。
Applications
Immediate Applications
自动驾驶辅助
模型主动调整视角识别道路和障碍物，提高行车安全。
机器人导航
机器人通过主动视野调整，探索未知环境，实现自主导航。
Long-term Vision
智能交互系统
实现更自然的人机交互，模型能主动理解环境变化，提供个性化服务。
Abstract
Active perception, a crucial human capability, involves setting a goal based on the current understanding of the environment and performing actions to achieve that goal. Despite significant efforts in evaluating Multimodal Large Language Models (MLLMs), active perception has been largely overlooked. To address this gap, we propose a novel benchmark named ActiView to evaluate active perception in MLLMs. We focus on a specialized form of Visual Question Answering (VQA) that eases and quantifies the evaluation yet challenging for existing MLLMs. Meanwhile, intermediate reasoning behaviors of models are also discussed. Given an image, we restrict the perceptual field of a model, requiring it to actively zoom or shift its perceptual field based on reasoning to answer the question successfully. We conduct extensive evaluation over 30 models, including proprietary and open-source models, and observe that restricted perceptual fields play a significant role in enabling active perception. Results reveal a significant gap in the active perception capability of MLLMs, indicating that this area deserves more attention. We hope that ActiView could help develop methods for MLLMs to understand multimodal inputs in more natural and holistic ways.