An Embodied Generalist Agent in 3D World

Recent progress in multi-modal AI, exemplified by models like CLIP and ALIGN, has enabled significant advances in image-text understanding. However, these models are predominantly 2D-focused, limiting their ability to interpret complex 3D environments crucial for embodied tasks. Prior works such as Scan2Cap, 3DSSG, and 3D Visual-Language models have made strides in scene understanding, but data scarcity, lack of unified architectures, and limited task scope hinder real-world applicability. The emergence of large pre-trained language models (e.g., GPT, Vicuna) offers new opportunities for integrating language and perception. Nonetheless, adapting these models for embodied 3D understanding remains challenging due to the complexity of scene relations, object attributes, and the need for multi-task learning. This context motivates the development of scalable, multi-modal embodied agents capable of comprehensive scene comprehension and interaction.

Despite advances, existing models struggle with understanding and acting within real 3D environments due to limited data, fragmented architectures, and inability to generalize across tasks. The core challenge is to develop a unified system that can perceive complex 3D scenes, ground language descriptions, and execute embodied actions robustly. Achieving this requires large-scale, high-quality 3D VL datasets, effective multi-modal fusion techniques, and flexible training strategies that support diverse tasks like captioning, question answering, navigation, and manipulation. Overcoming these barriers is essential for deploying AI in real-world scenarios such as robotics, AR, and virtual assistants, where understanding spatial relations and executing actions are critical.

This work introduces a scene graph-guided data generation pipeline assisted by LLMs, enabling scalable creation of high-quality 3D VL datasets. It proposes a multi-modal transformer architecture that combines 2D image encoders, 3D point cloud encoders, and spatial transformers to model object relations and scene context. The two-stage training scheme—first aligning visual and language modalities, then instruction tuning—supports multi-task learning across diverse embodied tasks. The integration of LoRA fine-tuning preserves pre-trained knowledge while adapting to specific tasks. These innovations collectively enable LE0 to understand and act in complex 3D environments with high accuracy and flexibility, surpassing prior models in both understanding and generalization.

• �� Data collection: Aggregate object and scene data from Objaverse, ScanNet, 3RScan, MP3D, and CLIPort.
• �� Data generation: Use scene graphs as prompts, employing LLMs (e.g., ChatGPT) with object-centric Chain-of-Thought (O-CoT) prompting to produce scene descriptions, questions, dialogues, and task instructions.
• �� Data refinement: Apply filtering, logical reasoning checks, and manual corrections to ensure high quality and reduce hallucinations.
• �� Model architecture: Encode egocentric 2D images with OpenCLIP ConvNext, process 3D point clouds via PointNet++ and spatial transformers, and encode text with a tokenizer. Combine these into a unified token sequence.
• �� Training: First, perform VL alignment with object-level and scene-level captioning tasks. Then, conduct instruction tuning on diverse tasks like QA, navigation, and manipulation, optimizing with cross-entropy loss.
• �� Fine-tuning: Use LoRA to adapt the large language model, maintaining pre-trained knowledge while enabling task-specific learning.
• �� Inference: Generate responses via autoregressive decoding, supporting multi-modal inputs and multi-task outputs.

The evaluation spans datasets like Scan2Cap, ScanQA, SQA3D, MP3D, and CLIPort, assessing captioning, QA, embodied reasoning, navigation, and manipulation. Metrics include CIDEr, BLEU, METEOR, accuracy, and success rate. Baseline comparisons involve task-specific and generalist models. Ablation studies examine data diversity, model size, and training strategies. Qualitative analyses demonstrate scene grounding, multi-turn dialogue, and action execution. The experiments validate LE0’s superior performance and robustness across tasks, with detailed hyperparameter tuning and cross-scenario testing.

LEO achieves 72.4 CIDEr on Scan2Cap, outperforming previous models by over 10%. In 3D QA, accuracy exceeds 50%, surpassing prior baselines. Success rates in robotic manipulation reach 86.2%, demonstrating strong generalization. Ablation shows larger models and diverse data significantly boost performance. Multi-task fine-tuning enhances understanding in complex scenarios, with qualitative results confirming flexible, context-aware responses and actions. These outcomes establish LE0 as a new state-of-the-art in embodied 3D AI.

LE0 can be deployed in autonomous robots for household, industrial, or service tasks, enabling navigation, object manipulation, and scene understanding. It supports virtual assistants in AR/VR environments, providing natural multi-modal interactions. The model’s multi-task capabilities facilitate scalable deployment in complex real-world settings, reducing human effort and increasing efficiency. Long-term, LE0’s framework could evolve into fully autonomous agents capable of continuous learning and adaptation in dynamic environments, transforming industries like manufacturing, healthcare, and entertainment.

The reliance on large-scale, high-quality 3D VL datasets limits scalability due to data collection costs. Model performance degrades in highly dynamic or unstructured environments, indicating a need for robustness improvements. Computational costs are high, with 7B parameters requiring significant resources, hindering real-time deployment. The current focus on simulated datasets may not fully translate to real-world scenarios, necessitating further domain adaptation and robustness testing. Future work should address data efficiency, model compression, and real-world validation to enable broader adoption.