GeBDA: Building Damage Assessment as Text-Based Sequence Prediction
This paper introduces GeBDA, a sequence prediction approach using the Gemma model for end-to-end building damage assessment from bi-temporal satellite images, achieving competitive localization and classification.
Key Findings
Methodology
The approach formulates damage assessment as a sequence generation task, leveraging the pre-trained Gemma model. Building polygons are converted into discretized bounding box coordinates and damage labels, serialized into text sequences. The model inputs include pre- and post-disaster images and a textual prompt, producing a sequence that encodes all buildings' locations and damage grades. Training employs cross-entropy loss with teacher forcing; inference uses greedy decoding and Hungarian matching for evaluation. This end-to-end framework avoids the drift and hallucination issues typical in dense localization, enabling joint detection and classification within a single pass.
Key Results
- On the xBD dataset, the model achieved a boundary box F1-score of 78.34 and damage classification F1-score of 72.49, nearing pixel-based methods like UNet (83.46/66.50). On Bright, F1 scores were 81.23 (boxes) and 51.56 (damage), demonstrating strong optical modality performance, but SAR modality performance was limited due to visual encoder constraints.
- Joint training (GeBDA*) did not significantly improve results, indicating modality gaps affect generalization, especially across different data types. The model's localization accuracy was validated via rasterized metrics, reaching nearly oracle-level performance.
- The approach effectively reduces complex dense detection to sequence prediction, showing promise for scalable, automated damage assessment in disaster scenarios.
Significance
This work advances disaster response by providing a unified, end-to-end framework that simplifies building localization and damage classification. Moving away from traditional dense detectors, it leverages general pre-trained models for rapid deployment, addressing long-standing challenges in dense target detection and multi-modal data fusion. The sequence prediction paradigm opens new avenues for scalable, flexible, and interpretable remote sensing applications, with potential impacts on urban planning, environmental monitoring, and emergency management.
Technical Contribution
The paper introduces a novel encoding scheme that discretizes bounding box coordinates into text tokens, enabling the use of a general vision-language model for dense object localization. It supports dynamic input resolutions, integrates a matching-based evaluation, and demonstrates end-to-end training without auxiliary detection modules. These innovations facilitate joint localization and damage assessment, reducing reliance on specialized detectors and enabling scalable multi-object sequence generation.
Novelty
This is the first application of a general-purpose VLM, specifically Gemma, to perform joint building localization and damage grading directly from satellite imagery. The key innovation lies in converting spatial bounding boxes into textual sequences, bypassing the limitations of dense detection drift and hallucination. This approach represents a significant departure from prior methods that rely on external detection modules or simplified proxies, establishing a new paradigm for multimodal disaster assessment.
Limitations
- The model struggles in densely packed regions, often over-segmenting large buildings or missing small ones due to boundary ambiguity and sequence length constraints.
- SAR imagery remains challenging, as the frozen RGB-trained encoder cannot extract robust features from non-optical modalities, limiting multi-modal robustness.
- Sequence length caps at 200 buildings per patch, which may be insufficient in extremely dense urban environments, necessitating multi-scale or hierarchical approaches.
Future Work
Future efforts will focus on jointly fine-tuning the visual encoder for multi-modal robustness, integrating reinforcement learning to optimize spatial metrics like IoU, and scaling model size with parameter-efficient techniques such as LoRA. Additionally, expanding the model's generalization to diverse disaster types and multi-temporal data will be prioritized to enhance real-world applicability.
AI Executive Summary
Natural disasters demand rapid, accurate assessments of building damages to coordinate effective rescue and recovery efforts. Traditional methods rely heavily on manual interpretation or specialized detection models, which are often slow and limited in scalability. Recent advances in vision-language models (VLMs) have shown promise in multi-modal understanding, but their application to dense, multi-object localization tasks like building damage assessment remains challenging.
This paper introduces GeBDA, a novel framework that leverages the open-source Gemma model to transform the damage assessment task into a sequence prediction problem. By discretizing building boundary polygons into quantized bounding box coordinates and concatenating them with damage labels into textual sequences, the model directly generates comprehensive damage maps from bi-temporal satellite images. This end-to-end approach bypasses the complex multi-stage pipelines typical of dense detection, reducing drift and hallucination issues.
The core innovation lies in encoding spatial and semantic information as text tokens, allowing the model to perform joint localization and damage classification in a single pass. The architecture combines a Transformer backbone with a flexible vision encoder supporting dynamic input resolutions, trained with standard cross-entropy loss. Evaluation on the xBD and Bright datasets demonstrates that GeBDA achieves near state-of-the-art performance, with boundary box F1-scores of 78.34 and 81.23, respectively, and damage classification F1-scores approaching 72.49.
Despite promising results, the model faces limitations in densely packed urban environments and with SAR imagery, highlighting areas for future improvement. The authors plan to enhance multi-modal robustness, incorporate reinforcement learning for spatial metrics, and scale the model size for broader disaster scenarios. Overall, this work paves the way for scalable, automated building damage assessment, offering a new paradigm that combines the strengths of pre-trained VLMs with innovative sequence encoding strategies, promising significant impact on disaster response and urban monitoring.
Deep Dive
Abstract
Conventionally, Building Damage Assessment (BDA) is tackled either with dedicated network architectures or by fine-tuning geospatial image foundation models. In this work, we ask whether a general-purpose Vision-Language Model (VLM) can localize buildings and grade their damage through autoregressive sequence generation alone. We cast BDA as predicting a variable-length set of bounding boxes, each specified by its coordinates and a damage label. Our preliminary implementation, based on the open Gemma model, achieves promising damage mapping results from only bi-temporal satellite images and a suitable text prompt.