Geometry Reinforced Efficient Attention Tuning Equipped with Normals for Robust Stereo Matching
GREATEN integrates surface normals with efficient sparse attention to enhance cross-domain stereo matching, reducing errors by over 30%.
Key Findings
Methodology
GREATEN employs dual encoders to extract image and normal features, fused via Gated Contextual-Geometric Fusion (GCGF). It incorporates Specular-Transparent Augmentation (STA) to improve robustness against non-Lambertian reflections. Sparse attention modules (SSA, SDMA, SVA) optimize global feature extraction while reducing computational load. Trained solely on synthetic SceneFlow data, it achieves superior zero-shot generalization on ETH3D, KITTI, and other benchmarks, reducing errors by over 30%. The framework effectively combines geometric invariants with texture cues, enabling robust performance across diverse real-world scenarios.
Key Results
- On ETH3D, error decreases by 30%, achieving state-of-the-art performance. In non-Lambertian regions of Booster, error drops by 8.5%. On KITTI-2015, error reduces by 14.1%. The model runs 19.2% faster than previous GREAT-IGEV, supporting high-resolution (3K) inference with disparity range up to 768, demonstrating both efficiency and accuracy.
- Across five benchmarks, trained only on synthetic data, GREATEN surpasses VFM-enhanced methods, showing excellent cross-domain robustness. Ablation studies confirm the importance of GCGF, STA, and sparse attention modules, with significant gains in occluded and reflective regions.
- The proposed sparse attention mechanisms maintain global context modeling while lowering computational costs, enabling real-time high-res disparity estimation in complex scenes.
Significance
This work addresses the longstanding challenge of synthetic-to-real domain transfer in stereo matching, leveraging geometric invariants to overcome texture ambiguities. It advances the field by enabling models trained solely on synthetic data to perform reliably in real-world applications such as autonomous driving and 3D reconstruction. The integration of efficient sparse attention mechanisms paves the way for deploying high-accuracy stereo models in resource-constrained environments, marking a significant step toward industrial-scale deployment of robust depth estimation systems.
Technical Contribution
The key innovation lies in incorporating surface normals as domain-invariant geometric priors, fused with image features via GCGF to suppress texture ambiguities. STA introduces texture perturbations during training, enhancing model robustness against non-Lambertian effects. Sparse attention modules (SSA, SDMA, SVA) replace dense attention, drastically reducing computational overhead while preserving global context. The framework achieves superior generalization without reliance on Vision Foundation Models, representing a new paradigm in cross-domain stereo matching.
Novelty
This is the first comprehensive framework systematically integrating surface normals with sparse attention mechanisms for robust cross-domain stereo matching. Unlike prior works focusing solely on texture-based features or domain adaptation, GREATEN leverages geometric invariants to address ill-posed regions. The combined use of STA and sparse attention for efficiency and robustness is a novel contribution, enabling zero-shot generalization from synthetic to real data across multiple challenging benchmarks.
Limitations
- Despite improvements, the model still struggles with extreme non-Lambertian reflections and severe occlusions, indicating room for better geometric modeling. The reliance on synthetic training data limits fine-grained detail recovery. Sparse attention, while efficient, may face challenges in ultra-large disparity ranges beyond 768. Further integration with real data or self-supervised learning could enhance performance in complex environments.
Future Work
Future directions include incorporating multi-modal data such as LiDAR or radar to improve geometric understanding, exploring self-supervised training to reduce reliance on synthetic data, and extending sparse attention mechanisms to handle larger disparity ranges. Additionally, refining the model's ability to handle extreme lighting and reflective conditions will be crucial for real-world deployment in autonomous systems.
AI Executive Summary
Stereo matching is a fundamental task in computer vision, enabling the reconstruction of 3D scenes from stereo image pairs. Despite rapid progress with deep learning models like GCNet, PSMNet, and RAFT-Stereo, these methods often struggle to generalize across different domains, especially when faced with textureless, reflective, or occluded regions. Traditional approaches heavily depend on rich textures, which can be misleading in real-world scenarios, limiting their robustness. This challenge is particularly critical for applications such as autonomous driving, where environmental variability is high.
To address this, the authors propose GREATEN, a novel framework that integrates surface normals—geometric cues invariant across domains—into the stereo matching pipeline. The core components include the Gated Contextual-Geometric Fusion (GCGF) module, which adaptively fuses image textures and normal-based geometric features, suppressing ambiguous textures. The Specular-Transparent Augmentation (STA) strategy introduces controlled texture perturbations during training, simulating real-world non-Lambertian effects, thus improving robustness. Additionally, the framework employs sparse attention mechanisms (SSA, SDMA, SVA) that efficiently capture global context while significantly reducing computational costs.
Training solely on synthetic SceneFlow data, GREATEN demonstrates remarkable zero-shot generalization capabilities. It reduces errors by over 30% on ETH3D, 8.5% on the Booster dataset, and 14.1% on KITTI-2015, outperforming many VFM-augmented models. The sparse attention modules enable high-resolution inference (up to 3K) with disparity ranges up to 768, supporting real-time deployment. These results highlight the potential of combining geometric invariants with efficient attention mechanisms to overcome longstanding domain adaptation challenges in stereo matching.
Overall, this work marks a significant step toward robust, efficient, and generalizable depth estimation models, with promising implications for autonomous systems, 3D mapping, and robotic perception. Future research will likely focus on multi-modal fusion, self-supervised learning, and handling more extreme environmental conditions, further bridging the gap between synthetic training and real-world application.
Deep Dive
Abstract
Despite remarkable advances in image-driven stereo matching over the past decade, Synthetic-to-Realistic ZeroShot (Syn-to-Real) generalization remains an open challenge. This suboptimal generalization performance mainly stems from cross-domain shifts and ill-posed ambiguities inherent in image textures, particularly in occluded, textureless, repetitive, and non-Lambertian (specular/transparent) regions. To improve Synto-Real generalization, we propose GREATEN, a framework that incorporates surface normals as domain-invariant, object-intrinsic, and discriminative geometric cues to compensate for the limitations of image textures. The proposed framework consists of three key components. First, a Gated Contextual-Geometric Fusion (GCGF) module adaptively suppresses unreliable contextual cues in image features and fuses the filtered image features with normal-driven geometric features to construct domain-invariant and discriminative contextual-geometric representations. Second, a Specular-Transparent Augmentation (STA) strategy improves the robustness of GCGF against misleading visual cues in non-Lambertian regions. Third, sparse attention designs preserve the fine-grained global feature extraction capability of GREATStereo for handling occlusion and texture-related ambiguities while substantially reducing computational overhead, including Sparse Spatial (SSA), Sparse Dual-Matching (SDMA), and Simple Volume (SVA) attentions. Trained exclusively on synthetic data such as SceneFlow, GREATEN-IGEV achieves outstanding Syn-to-Real performance. Specifically, it reduces errors by 30% on ETH3D, 8.5% on the non-Lambertian Booster, and 14.1% on KITTI-2015, compared to FoundationStereo, Monster-Stereo, and DEFOM-Stereo, respectively. In addition, GREATEN-IGEV runs 19.2% faster than GREAT-IGEV and supports high-resolution (3K) inference on Middlebury with disparity ranges up to 768.