Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents
Introduces MissionBench, a benchmark for zero-shot evaluation of 22 MLLMs on 120 aerial long-horizon tasks, with success rates below 35%.
Key Findings
Methodology
This study builds a high-fidelity simulation platform using Unreal Engine 5 and Cosys-AirSim, creating 120 diverse missions across five real-world-inspired environments. Each mission is driven by a natural language instruction, requiring the model to autonomously plan routes, adjust viewpoints, and report outcomes. The evaluation employs a closed-loop framework, where models process egocentric RGB images and action histories to generate structured responses, including target localization, reasoning, and action primitives. No task-specific fine-tuning is performed, emphasizing zero-shot capabilities. Metrics include success rate, mission progress, oracle success, collision rate, and step efficiency. Experiments compare 22 models, revealing that the best achieves less than 35% success, far below human performance of 84.4%, highlighting the challenge of long-horizon embodied tasks. Larger models consistently outperform smaller ones, confirming the positive correlation between scale and capability. The analysis underscores that mission competence requires integrating perception, multi-step planning, and adaptive reasoning, extending beyond mere spatial perception.
Key Results
- The top model, Gemini 3.1 Pro, attains a success rate of 34.8% and a mission progress of 73.3%, significantly outperforming smaller models. Despite this, performance remains far below human benchmarks, indicating substantial room for improvement.
- Model scaling shows clear benefits: Gemini 3.1 Pro surpasses Flash Lite (success rate 24.0%) by over 10%, and GPT-5.4 outperforms Mini versions, validating the hypothesis that larger models possess stronger zero-shot embodied capabilities.
- Performance varies across task types, with reporting and manipulation tasks being the most challenging, emphasizing the need for multi-capability integration. The gap between models and humans suggests ongoing limitations in perception, planning, and control.
Significance
This work pioneers a comprehensive, task-level evaluation of off-the-shelf multimodal large models in aerial long-horizon tasks, revealing their current limitations and potential. It highlights that scale alone can enhance autonomous capabilities, but also raises concerns about reliability and safety in real-world deployment. The benchmark provides a standardized platform for future research, fostering development of more robust, capable autonomous systems for applications such as surveillance, disaster response, and environmental monitoring.
Technical Contribution
The paper introduces a novel simulation-based, mission-level evaluation framework that combines continuous 4-DoF control, natural language instructions, and multi-metric assessment. It demonstrates that large-scale models can perform complex tasks without fine-tuning, emphasizing the importance of model scaling. The framework enables detailed diagnosis of embodied reasoning failures, bridging perception, planning, and reporting in a unified setting, and sets a new standard for evaluating autonomous aerial agents.
Novelty
This is the first systematic, large-scale evaluation of off-the-shelf multimodal models on integrated, long-horizon UAV missions within a high-fidelity simulation environment. Unlike prior benchmarks focusing on navigation or isolated perception tasks, it assesses full mission execution, emphasizing the role of scale and multi-capability coordination. It advances understanding of zero-shot embodied AI, highlighting the importance of model size in complex, real-world tasks.
Limitations
- Models show limited robustness in complex, cluttered environments and are sensitive to perception errors, leading to failures in goal localization and task completion.
- Simulation-to-reality gap remains significant; real-world deployment may encounter environmental variability, sensor noise, and unforeseen scenarios.
- Current models struggle with long-distance planning and multi-object coordination, requiring further integration of reinforcement learning and fine-tuning for practical robustness.
Future Work
Future research should focus on integrating reinforcement learning and domain adaptation to improve real-world robustness. Enhancing multi-modal perception fusion, multi-object planning, and safety mechanisms will be crucial. Developing hybrid approaches combining pretraining with task-specific fine-tuning can further close the gap to human-level performance, enabling autonomous UAVs to operate reliably in complex, dynamic environments.
AI Executive Summary
This study introduces MissionBench, a comprehensive benchmark designed to evaluate the zero-shot, mission-level capabilities of multimodal large language models (MLLMs) in aerial robotics. Leveraging a high-fidelity simulation environment built on Unreal Engine 5 and Cosys-AirSim, the benchmark comprises 120 diverse tasks across five realistic scenarios, including reporting, inspection, manipulation, and patrol missions. Each task is initiated by a natural language instruction, requiring the model to autonomously plan routes, adjust viewpoints, and report outcomes within a continuous 4-DoF control framework. The evaluation employs a closed-loop system, where models process egocentric RGB images and action histories to generate structured responses, mimicking real-world autonomous decision-making.
The experimental results, involving 22 state-of-the-art models, reveal that the best-performing model achieves a success rate of only 34.8%, significantly below human performance at 84.4%. These findings underscore the substantial challenges faced by current models in executing complex, multi-step UAV missions without task-specific training. Notably, larger models consistently outperform smaller ones, confirming that scale positively correlates with embodied capabilities. The analysis highlights that successful mission execution demands the integration of perception, multi-step planning, and adaptive reasoning, extending beyond simple spatial awareness.
This work has broad implications for autonomous AI development, emphasizing that scale can unlock new capabilities but also introduces reliability and safety concerns. The benchmark sets a new standard for evaluating embodied AI in aerial contexts, guiding future research toward more robust, scalable, and safe autonomous systems capable of long-horizon tasks in real-world environments. Future directions include combining reinforcement learning, domain adaptation, and multi-modal fusion to enhance real-world applicability and safety, ultimately advancing autonomous UAVs toward fully reliable, intelligent agents.
Deep Analysis
Background
无人机自主导航与任务执行已成为AI研究的热点，早期工作如视觉SLAM、路径规划算法（如A*、RRT）解决静态环境中的路径问题。随着深度学习的发展，深度强化学习（如DQN、PPO）被引入提升自主决策能力。近年来，多模态大模型（MLLM）结合视觉与语言能力，推动无人机在复杂场景中的理解与交互。现有评估多集中于路径跟踪或目标搜索，缺乏对长时、多任务场景的系统性衡量。模型规模不断扩大，性能逐步提升，但在空间推理、多目标协调等方面仍存在瓶颈。
Core Problem
核心问题在于，现有模型难以在无人机长时任务中实现自主规划、视角调整和多任务协调，尤其在未微调条件下。长距离、多目标、多能力融合的复杂性使模型表现不稳定，亟需系统化的任务级评估框架以量化模型能力和不足。缺乏对多能力协同的全面理解限制了自主无人机的实际应用。
Innovation
本研究创新点包括：1）提出基于高仿真平台的任务级评估体系，支持连续4自由度控制和多指标评价；2）设计闭环评估框架，结合自然语言指令和模型推理能力，全面衡量模型表现；3）系统分析模型规模与性能关系，验证大模型在无人机长时任务中的潜力。这些创新突破了传统路径导航的局限，为自主无人系统的智能化提供新思路。
Methodology
- �� 构建高仿真环境：利用Unreal Engine 5和Cosys-AirSim，模拟五个真实场景，确保环境多样性和复杂性。
- �� 任务设计：每个任务由自然语言指令引导，涵盖报告、巡逻、操控和检测，目标对象在环境中预置。
- �� 输入设计：模型接收连续三帧egocentric RGB图像和动作历史，输出结构化响应，包括目标定位、推理、动作指令和完成标志。
- �� 控制机制：模型选择8个方向性动作（前、后、左右、上下、转向），每次动作支持连续调节距离或角度，支持精细控制。
- �� 评估指标：成功率、任务进展、最优成功率、碰撞率和步骤效率，全面衡量模型能力。
- �� 实验流程：在不同模型（开源和闭源）上进行三次重复，统计平均表现，分析模型规模与性能关系。
Experiments
采用五个高仿真环境，设计120个任务，分为训练集和测试集。模型无微调，使用API接口进行推理。评估指标包括成功率（SR）、任务进展（MP）、最优成功率（OSR）、碰撞率（CR）和步骤效率（Eff）。对比不同规模模型（如Gemini、GPT-5.4、Qwen系列）表现，分析模型规模对能力的影响。还进行了模型微调和不同任务类型的性能分析，验证模型在复杂场景中的表现差异。
Results
最大模型Gemini 3.1 Pro成功率为34.8%，明显优于其他模型，且模型规模越大，性能越好。模型在报告和操控任务中的成功率最低，反映多能力融合难度。模型表现远低于人类（成功率84.4%），但规模扩展带来显著提升，验证了大模型在无人机长时任务中的潜力。模型在不同环境和任务类型中的表现差异，揭示了多能力协调的关键挑战。
Applications
该平台可用于开发自主无人机系统，提升其在搜索、巡逻、检测等复杂任务中的能力。适合军事、安防、环境监测等行业，依赖高仿真训练和评估，减少实际部署风险。未来，结合强化学习和微调，将实现更鲁棒的自主操作。
Limitations & Outlook
模型在复杂环境中仍易受感知误差影响，表现不稳定。仿真环境与真实场景存在差异，实际应用中可能遇到环境变化和传感器噪声。此外，长距离、多目标任务的规划能力有限，未来需结合强化学习和微调优化模型性能。
Plain Language Accessible to non-experts
想象你在操控一架无人机，就像在玩一款复杂的飞行模拟游戏。你只给它一句话，比如“飞过去看看”，但没有告诉它具体怎么飞。它需要自己决定飞多远、转多快、看哪个方向，甚至在飞行中不断调整视角，找到目标后再报告。这个过程就像你在玩“寻宝”游戏，游戏里你要自己探索、判断、行动，直到找到宝藏。科学家用虚拟环境模拟这些任务，让无人机自己学习如何在复杂场景中飞行、观察和报告。结果显示，模型越大、越聪明，表现越接近人类，但还差得远，就像高手和新手的差距。这项工作帮助我们理解，未来的无人机可以更聪明、更自主，能在复杂环境中完成各种任务。
ELI14 Explained like you're 14
想象你在操控一架无人机，就像在玩一款超级复杂的飞行游戏。你给它一句话，比如“拍下那辆车的牌照”，但你没有告诉它具体怎么飞、怎么转、怎么观察。它必须自己决定飞多远、转多快、看哪个方向，甚至在飞行中不断调整视角，找到目标后再告诉你结果。这就像你在找丢失的宠物，只有一句提示，你得自己探索、判断、行动，直到找到它。科学家用虚拟的环境模拟这些任务，让无人机自己学习怎么在复杂场景中飞行、观察和报告。结果发现，模型越大、越聪明，表现越接近人类，但还差得远，就像高手和新手的差距。这个研究让我们知道，未来的无人机可以变得更聪明、更自主，能在复杂环境中完成各种任务，就像你在游戏中变成了真正的飞行员一样。
Abstract
Multimodal Large Language Models (MLLMs) are emerging as core reasoning modules for embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon embodied tasks from a single high-level instruction. We introduce MissionBench, a benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families. Agents must autonomously plan, navigate, and report outcomes using only egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for embodied AI.