Revisiting Cross-View Completion: Self-Supervised Pre-Training via Reconstruction Error Comparison
Gekko uses relative reconstruction error improvement to enhance 3D features without 3D labels, outperforming CroCo.
Key Findings
Methodology
Gekko employs three forward passes: (1) cross-view completion reconstructing the target from masked target and reference view, (2) masked autoencoding reconstructing from masked target only, and (3) predicting the pixel-wise relative error improvement between the two. The architecture is based on ViT, with an additional output channel for the relative error map. The training leverages the difference in reconstruction errors as a self-supervised proxy for co-visibility, requiring no depth or camera pose annotations. This multi-task setup enables the model to learn geometric cues directly from raw videos or image pairs, significantly improving downstream tasks like zero-shot correspondence, pose estimation, and point cloud regression.
Key Results
- On ETH3D, Gekko reduces encoder AEPE by 22% and decoder AEPE by 21%, surpassing CroCo, indicating stronger geometric feature learning.
- On ScanNet-1500, Gekko achieves up to 6× higher accuracy in relative pose estimation at strict thresholds and reduces point cloud regression error by 22%.
- Training directly from raw videos without 3D labels, Gekko generalizes well, with the extra channel serving as a robust co-visibility detector on unseen scenes, outperforming existing backbones.
Significance
This work addresses the core limitation of existing multi-view self-supervised methods that rely on depth or pose supervision. By introducing a novel error-based proxy for co-visibility, Gekko enables effective geometric learning from unannotated videos, broadening the applicability of self-supervised 3D vision. Its ability to learn from raw data reduces preprocessing costs and enhances scalability, paving the way for large-scale deployment in robotics, AR/VR, and autonomous navigation. The approach also offers insights into how error comparisons can serve as geometric cues, inspiring future research in unsupervised scene understanding.
Technical Contribution
Gekko's key innovation is the use of the relative reconstruction error improvement as a dense, self-supervised co-visibility indicator, integrated into a multi-task ViT framework. This design maintains architectural simplicity, only adding an output channel, and leverages the error difference as a soft supervision signal. The method enables training directly on raw videos with a simple stride curriculum, eliminating the need for explicit 3D annotations or preprocessing. It also demonstrates that this proxy significantly boosts downstream task performance, establishing a new paradigm for geometry-aware self-supervised learning.
Novelty
This is the first work to explicitly model the relative improvement in cross-view reconstruction error as a self-supervised proxy for co-visibility, bypassing the need for depth or pose labels. Unlike prior methods that rely solely on monocular cues or ground-truth annotations, Gekko leverages the inherent geometric information embedded in the error differences, leading to superior feature quality and generalization, especially in low-overlap scenarios.
Limitations
- The approach may struggle in scenes with heavy occlusion or repetitive patterns, where error signals become ambiguous. Its robustness in highly cluttered or dynamic environments remains to be validated.
- Training from raw videos is computationally intensive, requiring significant resources. The method's scalability to extremely large datasets or real-time applications needs further exploration.
- Extension to multi-view setups beyond pairs is non-trivial and requires additional research to define appropriate relative error proxies.
Future Work
Future directions include extending the framework to multi-view sequences, incorporating temporal consistency, and exploring adaptive weighting of the relative error prediction. Integrating this approach with reinforcement learning for active scene exploration and improving robustness in dynamic environments are also promising avenues. Additionally, optimizing training efficiency and applying the method to outdoor, large-scale datasets will further enhance its practical impact.
AI Executive Summary
Understanding the three-dimensional structure of scenes from visual data is fundamental for numerous applications, from robotics to augmented reality. Traditional methods rely heavily on annotated datasets with depth maps or precise camera poses, which are costly and limit scalability. Recent advances in self-supervised learning, such as CroCo, have leveraged multi-view cues through masked autoencoding and cross-view completion, but they face intrinsic limitations in non-overlapping regions where reference views provide little information.
This paper introduces Gekko, a novel self-supervised framework that overcomes these limitations by exploiting the relative improvement in reconstruction errors between cross-view completion and monocular autoencoding. By predicting this error difference at the pixel level, Gekko generates a dense, geometric proxy for co-visibility without requiring any 3D annotations. The architecture, based on Vision Transformers, remains largely identical to CroCo, with only an added output channel for the relative error map, ensuring a fair comparison.
Extensive experiments demonstrate Gekko’s superiority across multiple tasks. On ETH3D, it reduces the encoder AEPE by 22%, and on ScanNet-1500, it achieves up to six times higher accuracy in pose estimation at strict thresholds, outperforming CroCo by significant margins. Remarkably, Gekko can be trained directly from raw videos using a simple stride curriculum, eliminating the need for complex 3D preprocessing, and still matching the performance of models trained on curated datasets.
The core innovation lies in leveraging the error difference as a soft geometric signal, enabling the model to learn scene structure effectively without supervision. This approach broadens the scope of self-supervised 3D vision, making it more scalable and applicable to real-world, unstructured data. While challenges remain in scenes with heavy occlusion or repetitive patterns, the results open promising pathways for large-scale, annotation-free 3D scene understanding, with potential impacts spanning autonomous navigation, AR, and beyond. Future work aims to extend multi-view capabilities, incorporate temporal dynamics, and optimize training efficiency, further pushing the boundaries of unsupervised 3D learning.
Deep Dive
Glossary
Reconstruction Error (重建误差)
衡量模型重建目标图像与预测图像差异的指标，反映重建质量。技术上为像素差的平方和。
用于评估CroCo和MAE在不同区域的重建效果。
Co-visibility (共视性)
两个视角中同时可见的场景区域，反映视图间的几何关系。技术上通过深度或相机参数确定。
Gekko利用误差改善预测共视区域。
Masked Autoencoder (掩码自编码器)
一种自监督学习模型，通过掩码部分输入，重建完整图像。技术上为Transformer架构。
CroCo和Gekko的基础架构。
Relative Error Improvement (相对误差改善)
交叉视图补全误差与单视图自编码误差的比值，作为几何关系的代理。
Gekko的核心创新，用于无监督共视性检测。
Vision Transformer (ViT,视觉变换器)
基于Transformer的图像处理架构，将图像划分为块，进行自注意力计算。
Gekko架构基础。
Open Questions Unanswered questions from this research
- 1 如何在极端遮挡或动态场景中增强误差代理的鲁棒性仍未解决，未来需要结合时间信息或多尺度特征。
- 2 多视图扩展的理论和实践尚不成熟，尤其是在多目标、多遮挡环境中，如何定义和利用多视图误差改善是未解难题。
Applications
Immediate Applications
自主导航
利用Gekko从未标注视频中学习场景几何，提升机器人和无人机的环境理解能力，降低标注成本。
三维重建
在缺乏深度标注的情况下，从视频中自动学习场景结构，支持虚拟现实和增强现实应用。
Long-term Vision
大规模无标注场景理解
实现无需人工标注的城市级或户外场景三维建模，推动智慧城市、自动驾驶等行业发展。
Abstract
Self-supervised pre-training via cross-view completion learns strong features for 3D vision from co-visible regions of image pairs. However, the reference view provides little information for reconstructing non-co-visible patches, implicitly yielding a monocular training signal in these regions. We introduce Gekko, which turns this limitation into a useful signal. The relative improvement of the cross-view reconstruction error over a masked-autoencoder error is a self-supervised proxy for co-visibility: large improvements indicate co-visible regions, negligible ones non-co-visible areas. Gekko is a network, trained from scratch, that jointly performs cross-view completion, masked autoencoding, and per-pixel prediction of this relative improvement, providing an additional binocular signal for all masked regions without any ground-truth 3D annotation. Under identical architectures and training data, Gekko consistently outperforms CroCo on zero-shot correspondence estimation, relative pose estimation, and pointmap regression, with up to 6 times higher accuracy at the strictest relative-pose threshold and a 22% drop in end-point error on ETH3D. The extra channel it learns is itself a strong co-visibility detector on unseen scenes, and Gekko's frozen features outperform released cross-view backbones of comparable or larger size. It can also be trained directly from raw videos with a simple stride-based curriculum, removing the cumbersome 3D preprocessing prior methods require while matching models trained on curated data. Code and pre-trained models are publicly available.