AeroReformer2: Spoken-Query Referring Segmentation for Aerial Images
AeroReformer2 employs a bilateral network with boundary-preserving visual paths and kernel linear cross-modal attention, achieving 62.09% mIoU in spoken-query remote sensing segmentation.
Key Findings
Methodology
This paper introduces AeroReformer2, a dual-branch architecture combining a boundary-preserving visual encoder with wav2vec 2.0 speech features. It utilizes a confidence-gated token memory (CGTM) and dual-scale kernel linear attention (SV-KLA) to efficiently fuse high-resolution visual details with variable-length speech tokens without dense affinity matrices. The model incorporates multi-accent speech synthesis and noise interference during training, creating a robust benchmark VoiceAeroRef from RISBench. The architecture leverages a boundary-aware visual path alongside a token-preserving speech encoding, enabling precise pixel-level segmentation under diverse acoustic conditions.
Key Results
- On the clean test set, AeroReformer2 with Swin-Base backbone achieves 62.09% mIoU and 68.22% oIoU, outperforming the strongest audio-adapted baseline by 5.38 and 2.08 percentage points respectively. Under hard interference conditions, it maintains a 54.09% mIoU, demonstrating robustness against rotor, wind, and mixed noise at three SNR levels.
- The model's multi-scale fusion and boundary refinement significantly improve small object detection and boundary accuracy, validated through ablation studies. The incorporation of diverse accents and noise conditions during training enhances generalization to real-world noisy environments.
- Experimental results confirm that the linear attention mechanism reduces computational complexity while maintaining high segmentation accuracy, making it suitable for high-resolution remote sensing applications.
Significance
This work advances the field of natural language interaction with remote sensing data, enabling intuitive voice-based target localization and segmentation. It addresses the long-standing challenge of integrating speech commands into pixel-level remote sensing analysis, opening pathways for more accessible and efficient geospatial data interpretation. The proposed architecture balances accuracy and efficiency, making it feasible for deployment in real-time systems such as UAVs and satellite platforms. The benchmark VoiceAeroRef provides a valuable resource for future research in robust multimodal remote sensing understanding, especially under noisy conditions.
Technical Contribution
The paper introduces a novel bilateral network architecture that combines boundary-aware visual encoding with token-preserving speech fusion, utilizing kernel linear attention to avoid dense affinity matrices. The confidence-gated token memory (CGTM) enhances robustness to speech variability and noise. The dual-scale SV-KLA mechanism enables efficient cross-modal fusion at high resolution, significantly reducing computational costs. The integration of wav2vec 2.0 for speech features and multi-accent synthesis during training ensures robustness to acoustic variability. These innovations collectively push the boundary of efficient, robust spoken-query segmentation in high-resolution remote sensing imagery.
Novelty
This is the first benchmark and model dedicated to full-sentence spoken-query referring segmentation for aerial images. It uniquely combines high-resolution visual encoding with variable-length speech tokens, employing linear attention to maintain efficiency. Unlike prior works limited to natural images or single-word queries, this study addresses the complexities of aerial imagery, including orientation, scale, and noise, providing a comprehensive evaluation framework. The introduction of a confidence-gated token memory and dual-scale attention mechanisms represents a significant step forward in multimodal remote sensing research.
Limitations
- Despite robustness, the model's performance degrades under extreme noise conditions, especially when speech recognition errors increase significantly, affecting segmentation accuracy.
- Training relies heavily on synthetic speech data, which may not fully capture real-world acoustic variability, potentially limiting deployment in uncontrolled environments.
- The architecture's complexity and high computational demand pose challenges for real-time deployment on resource-constrained devices, necessitating further optimization.
Future Work
Future research will focus on integrating end-to-end speech recognition with segmentation, reducing reliance on synthetic data, and improving robustness in real-world noisy environments. Efforts will also aim to optimize model efficiency for edge deployment, explore multi-modal fusion with environmental context, and extend benchmarks to include more diverse acoustic scenarios, ultimately enabling more natural and reliable voice-controlled remote sensing systems.
AI Executive Summary
Remote sensing imagery provides critical insights across urban planning, environmental monitoring, and disaster management. However, traditional pixel-level segmentation methods rely heavily on manual annotations or text-based commands, limiting natural interaction. This study introduces AeroReformer2, a novel bilateral network architecture that leverages spoken language as an intuitive interface for target localization and segmentation in aerial images. By integrating boundary-preserving visual encoding with a token-preserving speech fusion mechanism, the model achieves high accuracy while maintaining computational efficiency. The core innovation lies in employing kernel linear attention to avoid dense affinity matrices, enabling scalable high-resolution processing. The research further constructs VoiceAeroRef, a comprehensive benchmark dataset with diverse accents and noise conditions, to evaluate robustness in realistic scenarios. Experimental results demonstrate that AeroReformer2 attains 62.09% mIoU on clean data and sustains 54.09% under challenging interference, outperforming existing text-based methods by significant margins. This work marks a pivotal step toward natural, voice-driven remote sensing analysis, facilitating more accessible and efficient geospatial data interpretation. It opens avenues for deploying voice-enabled UAVs, satellites, and ground stations, transforming how humans interact with complex remote sensing systems. Future directions include end-to-end speech-to-segmentation models, real-world noise adaptation, and hardware-efficient architectures, promising a more intelligent and user-friendly remote sensing ecosystem.
Deep Analysis
Background
遥感图像的像素级理解在城市规划、环境保护、灾害响应等多个领域扮演着关键角色。早期研究主要依赖于手工标注和基于文本的目标识别，随着深度学习的兴起，语义分割技术逐步成熟，但仍受限于交互方式的单一性。近年来，跨模态学习和Transformer模型推动了自然语言引导的图像理解，但大多依赖于文本描述，缺乏对语音交互的支持。遥感目标具有尺度大、方向多样、细节复杂等特点，传统方法难以兼顾高精度和鲁棒性。现有遥感引导分割多采用全句表达，但未考虑在复杂环境中使用语音指令的实际需求，尤其是在噪声环境下的性能表现不足。
Core Problem
核心问题在于如何在遥感图像中实现基于完整句子语音指令的像素级目标分割。现有方法多采用文本输入，无法适应语音多样性和噪声干扰。遥感目标尺度大、方向多变，边界模糊，如何在保持高分辨率细节的同时实现鲁棒的跨模态融合，是技术难点。此外，密集的语音-视觉亲和矩阵计算成本高，限制了模型在高分辨率遥感图像中的应用。解决这一问题对于实现自然交互、提升遥感数据处理效率具有重要意义。
Innovation
本研究的创新点主要包括：1）提出结合边界保持路径与核线性跨模态注意力的双边网络架构，有效融合多尺度视觉信息与语音特征；2）引入信心门控Token记忆机制（CGTM），增强语音特征的鲁棒性和表达能力；3）设计双尺度语音-视觉核线性注意力（SV-KLA），避免生成密集亲和矩阵，显著降低计算复杂度；4）利用wav2vec 2.0进行语音特征编码，结合多样化口音与噪声干扰训练，提升模型在复杂环境中的表现。这些创新共同实现了遥感目标的高效、鲁棒引导分割。
Methodology
- �� 构建VoiceAeroRef语音遥感引导分割基准，采集多口音、多干扰条件下的语音数据，确保数据多样性。• 设计AeroReformer2架构，结合边界保持视觉路径与双尺度融合机制，利用核线性注意力实现高效跨模态融合。• 采用wav2vec 2.0提取语音特征，结合信心门控Token记忆机制（CGTM）增强鲁棒性。• 在训练中引入多样化口音和噪声干扰，采用随机采样策略确保模型泛化能力。• 设计多条件硬干扰评估，验证模型在噪声环境中的鲁棒性。• 通过不同尺度的融合路径，恢复细节边界，提升小目标检测能力。
Experiments
使用RISBench数据集，划分训练、验证、测试集，构建多干扰条件的硬干扰测试集。模型在清洁环境和噪声干扰条件下进行评估，指标为mIoU和oIoU。对比基线包括文本引导模型和传统遥感分割方法，进行消融实验验证各创新模块的贡献。超参数包括学习率、训练轮数、干扰强度等，确保模型在多场景下的鲁棒性。实验还分析了不同口音、干扰类型对性能的影响，验证模型的适应性。
Results
模型在清洁测试集达62.09%的平均交并比，硬干扰条件下仍保持54.09%，优于现有文本引导方法。引入多口音训练显著提升模型在不同语音变异下的表现。消融实验显示，核线性注意力和Token记忆机制对性能提升贡献最大。多尺度融合有效改善边界细节恢复，增强小目标检测能力。这些结果验证了模型设计的有效性和实用性，为遥感语音引导分割提供了新思路。
Applications
该技术可应用于无人机遥感、卫星监测、应急响应等场景，实现自然语音交互，提高操作效率。用户只需用自然语言描述目标，无需复杂标注或手动操作，极大简化遥感数据分析流程。未来可结合边缘计算，实现实时语音引导的遥感目标检测与分割，推动智能遥感系统的普及。
Limitations & Outlook
模型在极端噪声环境下仍存在性能下降，尤其在高干扰比例时误识别率升高。训练依赖大量合成语音数据，实际应用中真实语音环境可能带来额外挑战。模型结构复杂，对硬件资源要求较高，限制了在边缘设备上的部署。未来需优化模型结构，提升鲁棒性和效率，降低硬件门槛。
Plain Language Accessible to non-experts
想象你在一个大型工厂里工作，工厂里有许多不同的机器和操作员。每个操作员都用不同的语言和口音说话，有时还会遇到噪音，比如机器轰鸣声。你需要根据他们的声音找到对应的机器，然后对它们进行操作。传统的方法就像用手指点选机器，但这样很麻烦，也不方便。现在，工厂里装了一个智能助手，它能听懂各种不同的声音，甚至在噪音中也能准确识别目标。它会根据声音告诉你哪个机器需要注意，然后帮你找到对应的机器位置。这就像AeroReformer2一样，把语音变成一种特殊的信号，结合视觉信息，快速准确地找到遥感图像中的目标。这个助手还能在噪音很大的环境下工作，确保你不用担心声音变差带来的影响。它让工厂的工作变得更智能、更高效，节省了很多时间和精力。
ELI14 Explained like you're 14
想象你在学校的操场上玩捉迷藏，你的朋友用不同的声音告诉你“那边有个滑梯”或者“左边的树下有个球”。你要根据他们说的话，找到对应的东西。可是，有时候天气不好，风很大，声音变得模糊，你还怎么找到呢？这时候，如果你有一个超级聪明的耳机，它能听懂不同朋友的声音，无论多吵都能找到目标，就方便多了。AeroReformer2就像这个超级耳机，它能听懂遥感图像中的语音指令，帮你在复杂的环境中找到目标，比如在卫星图像中找到一座桥或者一辆车。它用一种特别的方法，把声音变成数字信号，和图像信息结合起来，即使在噪音很大的情况下，也能准确识别目标。这样，未来用语音指挥遥感设备就像和朋友聊天一样简单，不用手指点点点，直接说一声就能找到想要的东西。
Abstract
Spoken language offers a natural, hands-free interface for specifying an arbitrary target in dense remote-sensing imagery, yet existing referring remote-sensing image segmentation benchmarks accept only written expressions. To bridge this gap, we introduce \dataset, a spoken-query benchmark derived from RISBench that adds accent- and voice-diverse speech while preserving the original image, mask, and data splits. Its hard evaluation sets combine rotor, wind, and mixed interference with three signal-to-noise levels. We also propose \model, an efficient bilateral network that combines a boundary-preserving visual path with token-preserving speech encoding, kernel linear cross-modal attention, and a resolution refinement head. The design conditions visual features at two scales without materializing a dense speech--visual affinity matrix, then restores fine boundaries using high-resolution visual features. On the clean test split, \model with Swin-Base achieves 62.09\% mean intersection over union (mIoU) and 68.22\% overall intersection over union (oIoU), outperforming the strongest audio-adapted remote-sensing baseline by 5.38 and 2.08 percentage points, respectively. It retains the best hard-set mIoU at 54.09\%. To the best of our knowledge, this is the first benchmark and model study of full-sentence spoken-query referring segmentation for remote-sensing imagery. The code will be made publicly available.