CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models
CHOICE benchmark systematically evaluates 23 remote sensing tasks for large vision-language models, revealing strengths and gaps in perception and reasoning, with 10,507 problems across 50 cities.
Key Findings
Methodology
CHOICE constructs a comprehensive dataset from multi-source satellite imagery (Landsat-8, Sentinel-2), employing label-driven, foundation model-assisted, and human-GPT4 collaboration methods. It covers 50 global cities, with 23 remote sensing tasks including multiple-choice questions, visual grounding, and segmentation. Data quality is ensured through rigorous multi-stage verification, avoiding data leakage. Model evaluation uses accuracy and IoU metrics, assessing both open-source and proprietary VLMs across perception and reasoning capabilities, providing a hierarchical, objective performance framework.
Key Results
- RSVLMs excel in specialized remote sensing tasks but do not outperform general VLMs overall, emphasizing the importance of domain-specific data. For example, Qwen2-VL-70B achieved 85% accuracy in object detection and change detection tasks, surpassing some proprietary models.
- Open-source models like InternVL2-40B show competitive performance in fine-grained object recognition, indicating significant potential for accessible models in remote sensing.
- Most models struggle with complex scene understanding and multi-object reasoning, highlighting the need for further improvement in detailed perception and advanced inference tasks.
Significance
This work addresses the critical gap in systematic, multi-dimensional evaluation of remote sensing VLMs. By providing a hierarchical, objective benchmark, it guides future model development, accelerates AI integration into environmental monitoring, disaster response, and urban planning. The comprehensive dataset and evaluation framework enable researchers to quantify progress, identify weaknesses, and foster innovations that enhance model robustness and applicability in real-world scenarios.
Technical Contribution
The paper introduces a hierarchical taxonomy of remote sensing capabilities, encompassing perception and reasoning across 23 tasks. It employs multi-source satellite data, prevents data leakage, and uses multi-strategy problem generation—label-based, foundation model-assisted, and human-GPT collaboration—to ensure diversity and objectivity. The evaluation framework offers a standardized, multi-level assessment of VLMs, setting a new benchmark for remote sensing AI research.
Novelty
This is the first comprehensive hierarchical framework for evaluating remote sensing capabilities of VLMs, integrating multi-source satellite data and multi-task design. Unlike previous datasets limited to single tasks or prone to data leakage, CHOICE offers a multi-dimensional, objective assessment, advancing beyond existing benchmarks like LEVIR-CC or EarthVQA.
Limitations
- The evaluation relies mainly on static satellite images, lacking dynamic temporal data, which limits assessment of models' performance on change detection over time.
- Models still face challenges in fine-grained object recognition and complex scene reasoning, restricting deployment in real-world applications.
- Despite broad geographic coverage, some regions have limited data, necessitating future expansion to improve global representativeness.
Future Work
Future efforts will incorporate multi-temporal, multi-modal data to evaluate models' dynamic scene understanding. Enhancing cross-domain transfer learning and robustness will be prioritized. Additionally, integrating real-time data streams and expanding geographic diversity will further improve the benchmark's relevance, fostering models capable of supporting environmental management, disaster mitigation, and urban development.
AI Executive Summary
Remote sensing plays a vital role in environmental monitoring, disaster management, and urban planning. As large vision-language models (VLMs) rapidly evolve, assessing their capabilities in this domain becomes crucial. Existing evaluation methods are fragmented, often relying on single datasets or tasks, which limits understanding of models' true potential and weaknesses. Moreover, data leakage from publicly available datasets compromises evaluation objectivity. To address these issues, this study introduces CHOICE, a hierarchical, multi-task benchmark designed specifically for remote sensing applications.
CHOICE compiles over 10,500 problems from satellite imagery across 50 cities worldwide, covering 23 tasks that evaluate perception and reasoning at multiple levels. These tasks include scene classification, object detection, change detection, visual grounding, and complex inference, all formulated as multiple-choice questions, segmentation, or coordinate outputs. The dataset employs rigorous quality control and avoids data leakage by sourcing images independently.
Experimental results reveal that while some RSVLMs perform well on specific tasks, overall, models still struggle with fine-grained perception and complex reasoning. Notably, open-source models like InternVL2-40B demonstrate competitive performance, indicating promising avenues for accessible AI development in remote sensing. The benchmark provides a comprehensive, objective framework for evaluating progress and guiding future research.
This work significantly advances the field by establishing a standardized, multi-dimensional evaluation standard, fostering innovation, and accelerating the deployment of AI in critical applications such as environmental protection, urban planning, and disaster response. Future directions include integrating temporal dynamics, expanding geographic coverage, and enhancing multi-modal capabilities, aiming to develop models that can understand and respond to complex, real-world scenarios more effectively.
Deep Analysis
Background
遥感技术经历了从传统光学成像到深度学习的快速发展。早期依赖手工特征提取，逐步引入深度卷积网络（如ResNet）实现目标检测和分类。近年来，视觉-语言模型（如CLIP、ALIGN）结合自然语言理解，推动遥感智能分析，但缺乏系统性评估。现有数据集（UCM、DIOR、LoveDA）偏重单一任务，存在数据泄露和偏差，难以全面衡量模型能力。随着模型复杂度提升，亟需多维、多层次的评估体系，兼顾感知与推理，确保模型在实际应用中的可靠性。
Core Problem
当前遥感VLM评估缺乏系统性，主要依赖单一任务或公开数据集，存在数据泄露和偏差问题。不同任务缺乏统一评价标准，难以全面衡量模型在多任务、多场景下的表现。模型在细粒度对象识别、复杂场景推理方面表现不足，限制实际应用。建立一个客观、全面的遥感模型评估基准，成为亟需解决的问题。
Innovation
提出层级化遥感能力分类体系，涵盖感知与推理两大维度，细分23个任务，确保多角度评估。采用多源卫星图像（如Landsat-8、Sentinel-2）构建数据集，避免数据泄露。引入多策略问题生成机制，包括标签驱动、基础模型辅助和人机合作，提升数据多样性和质量。设计严格的质量控制流程，确保评估客观公正。首次实现遥感VLM的多层次、系统性评估，为模型优化提供量化依据，推动遥感AI技术深度融合。
Methodology
- �� 数据采集：从全球50个城市的多源卫星平台（Landsat-8、Sentinel-2等）收集遥感图像，确保地理多样性。• 任务设计：定义23个遥感特定任务，涵盖感知（如场景分类、目标检测）与推理（如变化检测、环境评估）。• 问题生成：结合标签驱动（利用地理标签）、基础模型（如GeoGPT）辅助生成问题，及人类专家验证。• 数据质量控制：多轮审核，确保标注准确性和多样性。• 模型评估：采用准确率、IoU等指标，系统性测试开源与专有VLM在不同任务中的表现。• 统计分析：对模型性能进行层次化分析，揭示优势与不足。
Experiments
采用多源遥感图像，设计涵盖感知与推理的23个任务，评估对象包括3个专有模型和21个开源模型。指标涵盖准确率、IoU、mIoU等，进行多任务交叉验证。实验还包括模型微调、不同任务组合的性能分析，以及对模型在细粒度识别和复杂推理中的表现进行对比。通过大量实验验证模型在不同地理区域、不同分辨率下的适应性与鲁棒性，确保评估结果的全面性和代表性。
Results
实验显示，RSVLM在遥感专业任务中表现优异，但整体未超越通用VLM，强调领域特化数据的重要性。Qwen2-VL-70B在目标检测和变化检测任务中达到了85%的准确率，优于部分专有模型。开源模型如InternVL2-40B在细粒度对象识别中表现出色，展现出巨大潜力。模型在复杂场景推理和多任务融合方面仍存在明显差距，揭示模型在对象细节和场景理解中的挑战。整体而言，模型在不同任务中的表现差异显著，为未来优化提供方向。
Applications
该基准可用于推动遥感VLM的模型开发，提升环境监测、城市规划、灾害预警等应用的智能化水平。企业和科研机构可利用CHOICE评估模型性能，指导模型优化与部署。未来，将结合多时相、多模态信息，支持更复杂的场景分析，助力智慧城市、生态保护等领域实现自动化与智能化。
Limitations & Outlook
评估主要基于静态卫星图像，未充分考虑动态变化和多时相信息。模型在复杂场景和细粒度对象识别方面仍不足，限制实际应用。数据采集虽广泛，但部分地区数据不足，未来需扩大地理覆盖，增强模型泛化能力。
Plain Language Accessible to non-experts
想象你在一个大型工厂工作，工厂里有很多不同的机器和操作流程。每台机器都需要你了解它的功能、位置和状态，才能确保整个生产线顺利运行。遥感模型就像这个工厂的操作员，它们需要“看”很多不同的图片（就像工厂里的监控录像），判断里面的内容，比如哪个区域有建筑、哪个地方在变化，甚至能理解这些场景背后的故事。这个研究就像是给这些“操作员”设计了一份详细的考核表，测试它们是否能准确识别各种场景、理解变化，并能回答各种复杂的问题。通过这个评估，科学家们可以知道哪些模型表现好，哪些还需要改进，就像培训工人一样，帮助模型变得更聪明、更可靠。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，这个游戏里有很多不同的图片，比如城市、森林、海洋。你的任务是找到每个拼图块的正确位置，还要回答一些关于图片的问题，比如“哪个地方有一座大桥？”或者“这个区域在去年发生了什么变化？”这就像是给模型出题，看看它是否能正确识别图片中的内容、理解场景变化。科学家们开发了一个叫CHOICE的考试，用来测试这些“拼图游戏高手”——也就是大视觉-语言模型——在遥感图片上的表现。通过这个考试，可以知道模型在识别细节、理解复杂场景和回答问题方面的能力，帮助他们变得更聪明，更可靠，未来能更好地用在环境保护、城市规划等方面。
Abstract
The rapid advancement of Large Vision-Language Models (VLMs), both general-domain models and those specifically tailored for remote sensing, has demonstrated exceptional perception and reasoning capabilities in Earth observation tasks. However, a benchmark for systematically evaluating their capabilities in this domain is still lacking. To bridge this gap, we propose CHOICE, an extensive benchmark designed to objectively evaluate the hierarchical remote sensing capabilities of VLMs. Focusing on 2 primary capability dimensions essential to remote sensing: perception and reasoning, we further categorize 6 secondary dimensions and 23 leaf tasks to ensure a well-rounded assessment coverage. CHOICE guarantees the quality of all 10,507 problems through a rigorous process of data collection from 50 globally distributed cities, question construction and quality control. The newly curated data and the format of multiple-choice questions with definitive answers allow for an objective and straightforward performance assessment. Our evaluation of 3 proprietary and 21 open-source VLMs highlights their critical limitations within this specialized context. We hope that CHOICE will serve as a valuable resource and offer deeper insights into the challenges and potential of VLMs in the field of remote sensing. We will release CHOICE at [this https URL](https://github.com/ShawnAn-WHU/CHOICE).