R3D: Quantitative 3D Spatial Reasoning for Egocentric Wearables
R3D constructs 3D scenes from egocentric RGB-D video and uses tool calls to achieve 73.5% accuracy in quantitative 3D reasoning tasks.
Key Findings
Methodology
R3D employs scene segmentation via SAM3, depth-based point cloud lifting, and multi-view consistency filtering to build accurate 3D scene representations. It then exposes geometric information through eight modular tools—such as get_object_volume and get_distance—to a large language model (Qwen3-VL 235B). This setup enables zero-shot reasoning on complex questions involving distances and volumes. Experiments demonstrate that R3D surpasses baselines like CuTR (61.9%) and Gemini 3 Flash (46.5%) with 73.5% accuracy, highlighting the effectiveness of scene construction combined with tool invocation for egocentric spatial tasks.
Key Results
- On R3D-Bench, R3D with Qwen3-VL 235B achieves 73.5% mean relative accuracy, outperforming all baselines. The approach significantly improves quantitative measurements in natural egocentric videos, especially in distance and volume questions. Ablation studies show scene construction and filtering are critical for performance. The model maintains robustness across partial observations and distant objects, with errors around 2%.
- Compared to models relying solely on RGB or depth features, R3D reduces measurement errors and improves consistency in complex scenes. It demonstrates strong generalization to challenging scenarios, validating the scene-based reasoning framework.
Significance
This work advances the field of wearable AI by enabling accurate, real-time quantitative spatial reasoning in natural environments. It addresses key limitations of prior benchmarks that relied on static scans or complete scene reconstructions, offering a practical solution for intelligent assistants, AR, and robotics. By integrating scene construction with tool-based reasoning, R3D provides a scalable, zero-shot approach that can be deployed on edge devices, paving the way for more autonomous and perceptive systems in daily life. The framework's ability to handle partial, egocentric observations marks a significant step toward human-like spatial understanding in AI.
Technical Contribution
The core innovation lies in combining scene segmentation (SAM3), depth-based point cloud lifting, and multi-view consistency filtering to generate accurate 3D scene models from egocentric video. These models feed into a modular tool system that exposes geometric properties—object positions, sizes, volumes, and distances—to large language models. The approach is model-agnostic, requiring no additional training, and leverages geometric filtering to handle partial observations. This design significantly improves quantitative reasoning capabilities in complex, real-world scenarios, setting a new standard for egocentric spatial AI.
Novelty
This is the first framework to perform end-to-end, zero-shot 3D spatial reasoning directly from egocentric RGB-D video, integrating scene construction with a tool-calling mechanism. Unlike prior methods that depend on full scene scans or static datasets, R3D dynamically builds scene representations from partial observations, addressing challenges like motion blur, occlusion, and distant objects. Its innovative use of multi-view consistency filtering and scalable tool interfaces distinguishes it from existing approaches, offering a practical, deployable solution for real-world wearable AI.
Limitations
- The system's accuracy diminishes in scenarios with severe occlusion, extreme distance, or poor depth quality, due to limitations in current depth estimation and scene reconstruction algorithms.
- Handling highly dynamic scenes with rapid object movement remains challenging, as the current framework primarily focuses on static or slow-moving environments.
- Dependence on pre-trained segmentation models like SAM3 and SAM3D may limit performance under adverse lighting or texture conditions.
Future Work
Future research will focus on integrating temporal modeling to handle dynamic scenes, improving depth estimation robustness, and enabling end-to-end training for scene reconstruction and reasoning. Expanding multimodal inputs, such as audio or tactile data, could further enhance spatial understanding. Additionally, optimizing computational efficiency for deployment on resource-constrained devices will be a key direction, aiming to realize truly autonomous, perceptive wearable systems.
AI Executive Summary
The rapid evolution of wearable devices equipped with RGB-D sensors and SLAM technology has opened new horizons for intelligent spatial reasoning in real-world environments. However, existing benchmarks and methods often fall short in addressing the complexities of natural egocentric videos, partial observations, and the need for precise quantitative reasoning. These limitations hinder the deployment of AI assistants capable of understanding and interacting with their surroundings at a human level.
This paper introduces R3D, a novel framework that constructs detailed 3D scene representations from egocentric RGB-D video streams. By leveraging state-of-the-art segmentation (SAM3), depth-based point cloud lifting, and multi-view consistency filtering, R3D generates accurate geometric models of objects and scenes. These models are then exposed to large language models via a set of eight spatial tools, enabling complex reasoning tasks such as measuring distances, volumes, and spatial relationships without requiring additional training.
The authors also present R3D-Bench, a comprehensive benchmark comprising 3,033 questions across 15 types, built on 57 videos from the Aria Digital Twin dataset. These questions simulate real-world scenarios, demanding precise quantitative answers in meters and liters. Experimental results show that R3D with Qwen3-VL 235B achieves 73.5% accuracy, outperforming previous methods by a significant margin. The framework demonstrates robustness in partial observation scenarios, distant objects, and challenging lighting conditions.
This work marks a significant step toward autonomous, perceptive wearable AI systems capable of real-time, accurate spatial understanding. It offers a scalable, zero-shot solution adaptable to various large language models and real-world applications, including AR, robotics, and assistive technologies. Despite current limitations in dynamic scene handling and extreme occlusion, future directions include integrating temporal modeling, multimodal inputs, and optimizing computational efficiency to realize fully autonomous spatial reasoning in everyday environments.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备食材，你只看到一部分，比如水壶和碗。你想知道倒完水后，碗里还剩多少水，或者水壶离水槽有多远。你只能用眼睛看到部分场景，比如只看到水壶的顶部或水槽的边角。这个系统就像一个聪明的助手，它能根据你看到的部分，推算出水壶的容量、碗里的水量，以及它们之间的距离。它用一些“工具”——比如测量距离、计算容量——来帮你解答复杂的问题。就像你用尺子和量杯一样，系统用数学和图像技术，把你看到的部分拼凑成完整的场景，然后告诉你答案。这让智能设备像人一样理解空间关系，帮你做出准确判断。它能在你只看到一部分的情况下，帮你理解整个房间或场景，甚至帮你做家务或找东西。
ELI14 Explained like you're 14
想象你在房间里玩捉迷藏，你只看到一部分房间，比如一张桌子和一个玩具熊。你想知道玩具熊离桌子有多远，或者桌子有多大。你不能看到整个房间，只能看到一部分。这个系统就像你的朋友，他能用眼睛看到的部分，猜出整个房间的布局。它用一种特别的方法，把你看到的部分变成3D模型，就像用积木搭房子一样。然后，它用数学计算，告诉你玩具熊离桌子有多远，桌子有多大。它还能帮你算出水壶还能倒出多少水，或者两个东西之间的距离。就像你用尺子和量杯一样，系统用电脑的“眼睛”和“脑袋”，帮你理解房间里的空间关系。这样，你可以用它玩游戏、找东西，甚至帮爸爸妈妈做家务啦！
Abstract
Quantitative 3D spatial reasoning from egocentric RGB-D video is a critical capability for next-generation wearable assistants. Yet existing benchmarks do not reflect the challenges of handling (1) natural egocentric video, (2) posed RGB-D video inputs, and (3) challenging quantitative 3D spatial reasoning Q&A. To fill this gap, we introduce R3D-Bench (Reasoning in 3D), a benchmark of 3,033 quantitative spatial reasoning questions across 15 types -- spanning multiple-choice, distance-based, and volumetric reasoning questions -- built on top of 57 egocentric video sequences from Aria Digital Twin. To set a strong baseline on this dataset, we introduce R3D, a model-agnostic spatial tool-calling framework. In contrast to existing approaches that directly embed 3D information into the model's input representation, R3D constructs a 3D scene from video using segmentation and depth-lifted object representations. It provides this information to an LLM through eight composable spatial tools. On R3D-Bench, R3D with Qwen3-VL 235B achieves 73.5% mean relative accuracy, substantially outperforming the best depth-enabled baseline (CuTR+Tools, 61.9%) and the best RGB-only baseline (Gemini 3 Flash, 46.5%).