Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

Recent advances in multimodal large language models (MLLMs), such as Liu et al. (2023a, 2024) and Bai et al. (2023), have demonstrated strong perception capabilities in image and video understanding. These models have been increasingly integrated into robotic control systems, with early efforts like TypeFly, PIVOT, and SPF focusing on visual grounding and waypoint pointing. However, these systems often depend on external detectors, fine-tuning, or specialized control modules, limiting their flexibility and generalization. As models scale up, researchers aim to leverage their reasoning and planning abilities directly for autonomous control, but challenges remain in protocol adherence, task termination, and multi-agent coordination. Existing benchmarks tend to evaluate isolated skills rather than integrated, closed-loop behaviors, creating a gap in understanding real-world applicability.

The core challenge is enabling multimodal large language models to perform end-to-end autonomous control in complex, multi-view, multi-target environments without relying on task-specific fine-tuning. Key issues include maintaining consistent action protocols, accurately declaring task completion, and coordinating multiple drones. Current systems often lack mechanisms for the model to self-assess success, leading to premature or delayed termination, which hampers deployment in real-world scenarios. Additionally, multi-view coordination remains problematic, with models prone to blindly copying coordinates across different perspectives, resulting in failures in multi-drone tasks.

This work introduces a prompt-driven, model-agnostic control architecture—DroneCATS-Agent—that defines four high-level actions declared solely via natural language prompts. It emphasizes the importance of a continuous, self-maintained protocol for task termination, integrating a verifier to assess the model’s self-declared success. Unlike prior systems that depend on external thresholds or specialized modules, this approach relies on the model’s own declarations, promoting autonomy and scalability. The evaluation across multiple model sizes and multi-drone scenarios provides new insights into the relationship between perception, protocol adherence, and control efficacy, representing a significant step forward in vision-language-action autonomous systems.

• �� Design DroneCATS-Agent with four core actions (go, rotate, think, finished), declared via prompts.
• �� Use a geometric controller to convert pixel and depth outputs into 3D setpoints, forming a closed-loop control system.
• �� Implement a verifier that evaluates the model’s self-declared completion, ensuring protocol adherence.
• �� Conduct experiments on two maps with nine models ranging from 2B to larger parameters, across four task types (approaching, tracking, searching, multi-drone commanding).
• �� Analyze success rates, failure modes, and protocol compliance through ablation studies and failure taxonomy.
• �� Compare models of different scales, emphasizing protocol adherence versus navigation performance.

Experiments involve two diverse maps—residential and campus—testing nine models on 80 single-drone episodes and 20 multi-drone episodes. Metrics include success rate, premature declaration rate, and protocol adherence. The evaluation also includes variance analysis and failure classification, focusing on how well models sustain the declared protocol. The experiments examine the impact of model size, initial target visibility, and target motion, providing comprehensive insights into the strengths and weaknesses of prompt-driven autonomous control. Additional ablation tests assess the effect of different prompt formulations and verifier thresholds.

Small models (~2B parameters) achieve a success rate of 65% in approaching tasks, outperforming some frontier models (~50%). However, they often prematurely declare success or fail to declare at all, indicating protocol adherence issues. In multi-drone scenarios, models tend to blindly copy coordinates across views, leading to coordination failures. Larger models, despite similar navigation abilities, exhibit worse protocol compliance, highlighting that the core challenge is sustaining a correct action protocol rather than perception. These findings emphasize the importance of designing models that can reliably self-assess and declare task completion.

The proposed framework can be applied to autonomous inspection, search-and-rescue, and environmental monitoring, where natural language commands and visual perception are sufficient for operation. Its plug-and-play design allows integration with existing drone platforms without extensive retraining, reducing deployment barriers. Future enhancements could include reinforcement learning for protocol persistence and multi-agent coordination, enabling scalable, reliable autonomous systems for real-world applications.

Models, especially smaller ones, struggle with maintaining a consistent protocol, often declaring success prematurely or not at all. Multi-view coordination remains problematic, with models copying coordinates blindly, leading to failures in multi-drone tasks. The current system's reliance on prompt-based declarations limits robustness in dynamic environments. Additionally, model scale constraints (2B parameters) restrict performance in complex scenarios, necessitating further research into scalable, efficient architectures that can sustain protocol adherence over extended operations.