GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
GraphDreamer uses scene graphs to generate disentangled 3D scenes, outperforming prior methods with CLIP score 0.3326.
Key Findings
Methodology
GraphDreamer constructs a scene graph from user input or ChatGPT, representing objects as nodes and relationships as edges. It encodes object attributes via identity-aware feature encoders and models geometry with signed distance fields (SDF). The framework employs score distillation sampling (SDS) to optimize 3D geometry and appearance, guided by multi-level text prompts derived from the scene graph. The model integrates global, node-wise, and edge-wise rendering, ensuring object disentanglement and relationship consistency. It includes geometric constraints like penetration avoidance and Eikonal regularization, enabling high-fidelity scene synthesis without manual scene graph creation.
Key Results
- In complex multi-object scenes, GraphDreamer achieves a CLIP score of 0.3326, surpassing Magic3D and MVDream. It accurately captures object attributes such as ‘ancient’, ‘glowing’, and relationships like ‘standing in front of’, with fully disentangled objects. Quantitative evaluations show significant improvements in object separation and relationship fidelity, validated through ablation studies. The automatic scene graph generation via ChatGPT enhances flexibility and reduces manual effort, enabling scalable scene synthesis.
- The approach demonstrates superior performance in preserving object attributes and inter-object relationships across diverse scenarios. It produces detailed, high-quality 3D scenes that align well with textual descriptions, with ablation results confirming the importance of scene graph guidance. The method generalizes effectively to unseen complex scenes, indicating robustness and scalability for practical applications.
Significance
This work advances 3D scene synthesis by integrating structured scene graphs, addressing the limitations of holistic text prompts in multi-object scenarios. It enables precise control over object attributes and relationships, crucial for applications in virtual reality, gaming, and robotics. By achieving fully disentangled objects within complex scenes, it opens new avenues for semantic scene understanding and editing. The automatic scene graph generation further democratizes scene creation, reducing reliance on manual annotations. Overall, GraphDreamer bridges the gap between structured scene understanding and high-fidelity 3D synthesis, marking a significant step toward intelligent, scalable virtual environment generation.
Technical Contribution
The paper introduces a novel framework combining scene graph-based conditioning with identity-aware object encodings and SDF geometry modeling. It innovates by integrating multi-level text prompts, object-specific feature encoders, and scene-level constraints within a score distillation sampling paradigm. The architecture supports automatic scene graph extraction via ChatGPT, enabling scalable, automated scene synthesis. The method ensures object disentanglement and accurate relationship modeling, outperforming existing state-of-the-art methods like Magic3D and MVDream in fidelity and semantic accuracy. The approach also introduces a new scene decomposition strategy that maintains geometric consistency and prevents object interpenetration.
Novelty
This is the first work to incorporate scene graphs directly into 3D generative frameworks, moving beyond simple text prompts or bounding box constraints. The combination of identity-aware feature encoding, SDF-based geometry, and multi-level prompt guidance represents a novel paradigm for structured 3D scene synthesis. Unlike prior methods limited to single-object or non-interactive scenes, GraphDreamer handles complex multi-object interactions with explicit relationship modeling, providing a more flexible and semantically rich generation process.
Limitations
- The reliance on scene graph accuracy means errors in automatic scene graph generation can affect scene quality. Complex or ambiguous descriptions may lead to incomplete or inaccurate graphs.
- Computational costs are high due to multiple rendering passes and large neural networks, limiting real-time applications.
- Current relationship modeling is mostly static; capturing dynamic interactions or temporal changes remains challenging. Further, the model's scalability to very large scenes needs exploration.
Future Work
Future directions include improving automatic scene graph extraction, integrating dynamic scene modeling, and optimizing computational efficiency. Extending the framework to handle temporal sequences for dynamic scene synthesis and incorporating user-interactive editing tools are also promising. Additionally, exploring multi-modal inputs like videos or real images to guide scene generation could broaden practical applications.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
Imagine you’re in a big workshop where you want to build a detailed model of a city. Instead of just throwing random buildings and cars together, you first draw a map showing where everything should go and how they relate—like which building is next to which, or which car is parked in front of a shop. This map is like a scene graph, a special kind of plan that clearly shows all the parts and their relationships. Now, a smart robot uses this map to build the city in 3D. It carefully shapes each building and car, making sure they fit without overlapping or missing details. The robot uses a special kind of ‘drawing tool’ that understands the shape and position of each object, and it follows the map step by step. Because of this plan, the city looks just like you imagined, with everything in the right place and the right size. This way, creating complex scenes becomes easier, faster, and more accurate, just like following a detailed blueprint makes building a model much simpler.
ELI14 Explained like you're 14
Think about playing with a giant LEGO set. If you just start stacking bricks randomly, it’s hard to make a cool castle or spaceship. But if someone gives you a simple drawing or a list of what goes where—like ‘a tall tower next to a big gate, with a knight standing in front’—it’s much easier to build exactly what you want. Now, imagine a super-smart robot that can read your drawing or list and then build the scene in 3D for you. It looks at your plan, understands which parts are which, and then carefully shapes each piece, making sure everything fits perfectly and looks just right. This robot uses a special kind of map called a scene graph, which shows all the objects and how they relate. Thanks to this, you get a detailed, accurate 3D scene that matches your idea perfectly, without you having to do all the hard work of building it by hand. It’s like having a magic LEGO assistant that turns your ideas into real 3D models in no time!
Abstract
As pretrained text-to-image diffusion models become increasingly powerful, recent efforts have been made to distill knowledge from these text-to-image pretrained models for optimizing a text-guided 3D model. Most of the existing methods generate a holistic 3D model from a plain text input. This can be problematic when the text describes a complex scene with multiple objects, because the vectorized text embeddings are inherently unable to capture a complex description with multiple entities and relationships. Holistic 3D modeling of the entire scene further prevents accurate grounding of text entities and concepts. To address this limitation, we propose GraphDreamer, a novel framework to generate compositional 3D scenes from scene graphs, where objects are represented as nodes and their interactions as edges. By exploiting node and edge information in scene graphs, our method makes better use of the pretrained text-to-image diffusion model and is able to fully disentangle different objects without image-level supervision. To facilitate modeling of object-wise relationships, we use signed distance fields as representation and impose a constraint to avoid inter-penetration of objects. To avoid manual scene graph creation, we design a text prompt for ChatGPT to generate scene graphs based on text inputs. We conduct both qualitative and quantitative experiments to validate the effectiveness of GraphDreamer in generating high-fidelity compositional 3D scenes with disentangled object entities.