Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers

Recent advances in computer vision have seen the rise of transformer-based models like DETR, which revolutionized 2D object detection by eliminating the need for heuristic post-processing. These models leverage self-attention mechanisms to produce object-level embeddings directly from images, achieving state-of-the-art performance on benchmarks like COCO. Prior research primarily focused on their detection accuracy, robustness, and extension to tasks like segmentation and pose estimation. However, the question of whether these models implicitly learn 3D scene understanding remains largely unexplored. Some studies have shown that dense feature maps in vision transformers encode physical scene cues, but target-level embeddings—key to object-centric reasoning—have not been systematically analyzed for 3D information. This gap limits our understanding of the models’ internal representations and their potential for unsupervised spatial reasoning. Addressing this, the current work investigates the extent to which object embeddings in pre-trained detection transformers encode depth and 3D position, providing insights into their geometric comprehension.

The core challenge is determining whether pre-trained 2D detection transformers, trained solely on 2D annotations, inherently encode 3D spatial information such as depth and object location. This is crucial because explicit 3D supervision is costly and limited in scope, yet many applications—autonomous driving, robotics, AR—demand accurate 3D understanding. If models can implicitly learn these cues, it could reduce reliance on expensive 3D annotations and enable more scalable perception systems. The difficulty lies in quantifying this implicit knowledge, as the embeddings are high-dimensional, complex, and not directly interpretable. The problem becomes one of probing the latent space to assess how well it encodes 3D attributes, and whether this encoding is linear or requires complex non-linear mappings.

The main innovations include: 1) Developing a systematic probing framework combining linear and MLP classifiers to analyze object-level embeddings for depth and position; 2) Applying this framework across multiple transformer variants trained without 3D supervision, revealing their latent 3D understanding; 3) Demonstrating that non-linear probes significantly outperform linear ones, indicating complex encoding relationships; 4) Showing that even lightweight models retain some 3D cues, though weaker than larger models. This comprehensive analysis uncovers the hidden geometric knowledge within models, challenging assumptions about the necessity of explicit 3D training.

• �� Select multiple pre-trained detection transformer models (DETR, Deformable DETR, Conditional DETR, RT-DETR, LW-DETR). • Extract object embeddings from the last decoder layer, paired with ground-truth or pseudo-labels for depth and 3D position. • Construct datasets by passing images through models, filtering detections based on objectness thresholds, and associating embeddings with corresponding 3D labels. • Train simple probes: linear (affine) and two-layer MLPs with ReLU activations, on frozen detector weights. • Evaluate probe performance using metrics like MAE, AbsRel, and accuracy thresholds. • Conduct experiments on datasets like Virtual KITTI 2 (outdoor) and NYUv2 (indoor), comparing across models and probe types. • Analyze the influence of model complexity and architecture on encoding capacity, and perform ablation studies to verify robustness.

Experiments involve applying probes to the last decoder layer of each model, assessing depth and 3D position prediction accuracy. Datasets include Virtual KITTI 2 and NYUv2, with ground-truth or predicted depth maps and 3D bounding boxes. Object embeddings are extracted for detected objects passing objectness thresholds. Probes are trained with Adam optimizer, batch size 512, for 1000 epochs, measuring MAE, AbsRel, and threshold accuracy. Results compare linear vs. non-linear probes, revealing the non-linear relationships. Additional experiments analyze model size effects, robustness under occlusion, and the impact of different backbone architectures. The setup ensures fair comparison across models and datasets, validating the hypothesis of implicit 3D encoding.

All models’ embeddings encode significant depth cues; non-linear probes outperform linear ones, with MAE for depth dropping below 0.55 on Virtual KITTI 2. and below 0.51 on NYUv2. For 3D position, errors are within 0.2 meters of specialized models like MonoDETR. Lightweight variants like RT-DETR v2 and LW-DETR show weaker performance, indicating a correlation between model complexity and spatial encoding. These results confirm that pre-trained 2D detectors have an inherent capacity for 3D understanding, even without explicit supervision, opening new avenues for unsupervised 3D scene perception.

Findings enable low-cost 3D perception in autonomous vehicles, robotics, and AR, leveraging existing 2D detection models without additional 3D annotations. They can be integrated into systems requiring depth and spatial localization, reducing reliance on expensive sensors. Long-term, this approach could lead to self-supervised models capable of comprehensive scene understanding, facilitating scalable deployment in real-world environments with minimal supervision.

The encoding strength diminishes under extreme viewpoints, occlusion, or clutter, limiting robustness. Current analysis focuses on depth and position; other attributes like pose or shape are unexplored. Model complexity influences encoding capacity, with lightweight models performing worse. The approach assumes accurate detection; failure cases in detection accuracy affect the probing results. Future work should address these limitations by incorporating multimodal data, dynamic scenes, and more complex geometric attributes.