ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

The evolution of multimodal large language models (MLLMs) has seen significant milestones, including GPT-4o, LLaVA, and PointLLM, which integrate vision and language understanding. These models excel in tasks like image captioning, visual question answering, and multimodal reasoning. However, their capabilities are predominantly confined to 2D modalities, limiting spatial comprehension. Recent advances in 3D content generation, such as Trellis and CRM, have demonstrated progress in shape synthesis, but often rely on separate modules or specialized representations like point clouds or meshes. The challenge remains in creating a unified model capable of seamless multi-task handling across text, images, and 3D assets, especially with a focus on high-fidelity, interactive editing, and real-time generation. The scarcity of large-scale, multi-task 3D datasets further constrains progress, impeding the development of truly 3D-native AI systems that can operate in complex spatial environments.

The core challenge addressed by this research is how to enable a single, end-to-end model to understand and generate 3D content within a multi-modal framework. Existing methods either treat 3D data as auxiliary or rely on multi-stage pipelines, which are inefficient and lack flexibility. The high dimensionality and topological complexity of 3D models make discretization and representation difficult, especially when aiming for scalable training and inference. Additionally, the limited availability of large, annotated 3D datasets hampers the training of comprehensive models capable of multi-task learning. Overcoming these bottlenecks is crucial for advancing spatial AI, enabling applications like real-time virtual environment editing, autonomous navigation, and immersive VR/AR experiences.

The key innovations include: 1) Developing a 3D VQVAE that encodes 3D models into a discrete token space, simplifying complex geometries into manageable units; 2) Constructing the 3D-Alpaca dataset with over 3.46 billion tokens, covering generation, understanding, and editing tasks, thus providing a rich resource for multi-task training; 3) Extending a pre-trained multimodal transformer (Qwen-2.5-vl-7B) with a dedicated 3D codebook, enabling the model to perform multi-modal, multi-task learning within a unified autoregressive framework. These innovations collectively address the limitations of prior models by integrating 3D content seamlessly into the multi-modal learning paradigm, supporting flexible input/output sequences and complex spatial reasoning.

• �� Encode 3D assets using a 3D VQVAE, which compresses voxel grids into 1024 discrete tokens, preserving shape and structural features.
• �� Render multi-view images of 3D assets, generate descriptive captions, and compile editing instruction pairs to build the 3D-Alpaca dataset.
• �� Fine-tune the Qwen-2.5-vl-7B model by adding a 3D codebook, freezing visual encoder parameters to retain image understanding while training the new 3D tokens.
• �� Represent all modalities as token sequences, enabling autoregressive prediction for tasks like text-to-3D, image-to-3D, captioning, and editing.
• �� Use a hybrid approach combining voxel-based encoding with mesh reconstruction via Rectified Flow for high-quality shape generation.
• �� Implement multi-task training with instruction tuning, leveraging the large dataset to improve model versatility and robustness.

The evaluation employs datasets such as Toys4K, Objaverse, and custom 3D-Alpaca subsets. Metrics include Frechet Distance, CLIP scores, and 3D understanding benchmarks. The training involves 15 epochs on 48 NVIDIA H100 GPUs, with hyperparameters like learning rate decaying from 5×10^-5 to 5×10^-6. Baselines include CRM, SAR3D, and Trellis, with comparisons across generation quality, geometric fidelity, and understanding accuracy. Ablation studies analyze the impact of dataset scale, model size, and discretization strategies. The experiments demonstrate the model’s capacity for high-fidelity, interactive 3D content creation and comprehension.

The model achieves Frechet Distance scores of 25.9 (text-to-3D) and 12.2 (image-to-3D), outperforming many prior methods. It maintains language reasoning abilities comparable to the base model, with a 49.72 score on 3D understanding benchmarks. Qualitative results show precise shape alignment with prompts, detailed geometry, and realistic textures. Interactive editing experiments successfully modify models based on natural language instructions, with 70k pairs validating its practical utility. The model also demonstrates robustness across diverse object categories and complex prompts, confirming its generalization capabilities.

This model can revolutionize virtual content creation, enabling artists and designers to generate and edit 3D assets via simple language commands. It supports rapid prototyping, immersive environment design, and interactive virtual reality applications. In robotics, it enhances spatial reasoning and environment understanding, facilitating autonomous navigation and manipulation. In industry, it accelerates digital twin development, virtual prototyping, and simulation tasks, reducing time and cost. The model’s multi-task ability also opens avenues for personalized virtual assistants, educational tools, and entertainment content generation, making 3D content creation more accessible and efficient.

Despite promising results, the model’s parameter size (7B) limits its capacity compared to larger models like ChatGPT-4o, affecting performance in highly detailed editing tasks. The current 3D editing dataset (70k pairs) is limited, constraining complex or high-fidelity modifications. Discrete token encoding via VQVAE can lead to loss of fine details, especially at higher resolutions. Computational costs remain high, hindering real-time deployment. Future work must focus on scaling model size, expanding datasets, and developing hybrid representations to improve detail fidelity and efficiency.