MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation
MMGDreamer introduces a dual-branch diffusion framework with a novel Mixed-Modality Graph, achieving high-precision geometry control in 3D indoor scene synthesis, outperforming state-of-the-art.
Key Findings
Methodology
This paper proposes MMGDreamer, a dual-branch diffusion model leveraging a novel Mixed-Modality Graph (MMG) that integrates textual and visual node features. The framework includes a visual enhancement module employing VQ-VAE to generate detailed node visual features, and a relation predictor based on GCN to infer missing relationships, ensuring scene coherence. Multi-modal features are encoded via CLIP, and the model is trained in two stages: first optimizing visual features and relation inference, then jointly refining layout and shape generation. The approach demonstrates superior control over object geometry and scene realism on the SG-FRONT dataset, outperforming existing methods.
Key Results
- On scene fidelity metrics (FID, KID), MMGDreamer surpasses baselines, reducing FID by 9% and KID by 33% in living rooms. Object geometry metrics (MMD) reach 0.22, significantly better than prior methods (>0.5). Multi-modal fusion enhances detail and layout accuracy, validating the model’s geometric control and diversity. Ablation studies confirm the importance of visual enhancement and relation prediction modules, with removal leading to degraded performance.
- Visual feature construction via VQ-VAE improves object shape detail; relation inference via GCN ensures scene layout consistency. The dual-branch diffusion process effectively generates both scene layout and object shape, yielding high-fidelity, controllable 3D scenes. Results demonstrate better diversity, detail, and geometric precision than previous state-of-the-art.
- Ablation results show that excluding either the visual enhancement or relation predictor increases FID and reduces scene coherence, confirming their critical roles. Overall, MMGDreamer achieves a new benchmark in controllable scene synthesis, especially in complex indoor environments.
Significance
This work advances the field of controllable 3D scene synthesis by integrating multimodal information and relation inference, addressing the limitations of text-only inputs. It enables more precise geometric control and scene realism, which are crucial for applications like virtual reality, interior design, and augmented reality. The proposed framework enhances user interaction, allowing flexible input modalities and producing highly coherent, detailed scenes. This paves the way for more intelligent, customizable virtual environments, significantly impacting both research and industry.
Technical Contribution
The core technical contributions include: 1) the design of the Mixed-Modality Graph supporting node features from text and images; 2) the visual enhancement module employing VQ-VAE for detailed node visual features; 3) the relation predictor based on GCN for inferring missing relationships; 4) a dual-branch diffusion model for synchronized layout and shape generation. These innovations enable high-precision geometric control and scene diversity, representing a significant step forward from previous single-modality or relation-dependent methods.
Novelty
This is the first work to introduce a flexible, multi-modal graph structure that allows nodes to contain both textual and visual features, with optional relationships. The combination of VQ-VAE-based visual feature construction, GCN-based relation inference, and a dual-branch diffusion process forms a novel pipeline that surpasses prior scene graph-based methods reliant solely on text. This integrated approach provides unprecedented control over object geometry and scene layout, marking a new paradigm in controllable 3D scene synthesis.
Limitations
- The model struggles with highly complex scenes involving intricate object interactions, due to limited training data diversity and relation inference accuracy.
- Computational costs remain high, especially for high-resolution scene generation, limiting real-time applications.
- Dependence on multimodal inputs can lead to degraded performance when inputs are noisy or incomplete, requiring further robustness improvements.
Future Work
Future directions include developing more efficient relation inference algorithms, incorporating multi-scale features for better detail preservation, and extending the framework to dynamic scenes and real-time interaction. Additionally, exploring reinforcement learning for layout optimization and integrating user feedback could further enhance controllability and practical deployment.
AI Executive Summary
Controllable 3D scene generation has become a vital component in virtual reality, interior design, and immersive environments. Existing methods often rely solely on textual descriptions, which limit the precision of object geometry control and scene coherence. To address these challenges, this paper introduces MMGDreamer, a novel dual-branch diffusion framework that leverages a Mixed-Modality Graph (MMG) to incorporate both textual and visual information at the node level. This structure allows for flexible user inputs, supporting various combinations of text and images, and optional relationships, significantly enhancing adaptability.
The core innovation lies in combining a visual enhancement module based on VQ-VAE with a GCN-based relation predictor. The visual module enriches node features with detailed visual representations derived from text embeddings, improving object shape fidelity. The relation predictor infers missing relationships, ensuring scene layout coherence. The entire system is trained in two stages: first optimizing visual features and relation inference, then jointly refining layout and shape generation via a dual-branch diffusion model.
Experimental results on the SG-FRONT dataset demonstrate that MMGDreamer outperforms state-of-the-art methods in both scene realism and geometric control. Metrics such as FID, KID, and MMD show significant improvements, especially in complex indoor scenes. Ablation studies confirm the importance of each module, with visual enhancement and relation inference providing substantial gains.
This work marks a significant step forward in controllable scene synthesis, enabling more precise, realistic, and user-friendly virtual environments. Its ability to handle flexible multimodal inputs and produce high-fidelity, geometrically accurate scenes opens new avenues for applications in VR, interior design, and beyond. Future research will focus on scalability, real-time interaction, and extending to dynamic scenes, pushing the boundaries of intelligent virtual space creation.
Deep Dive
Abstract
Controllable 3D scene generation has extensive applications in virtual reality and interior design, where the generated scenes should exhibit high levels of realism and controllability in terms of geometry. Scene graphs provide a suitable data representation that facilitates these applications. However, current graph-based methods for scene generation are constrained to text-based inputs and exhibit insufficient adaptability to flexible user inputs, hindering the ability to precisely control object geometry. To address this issue, we propose MMGDreamer, a dual-branch diffusion model for scene generation that incorporates a novel Mixed-Modality Graph, visual enhancement module, and relation predictor. The mixed-modality graph allows object nodes to integrate textual and visual modalities, with optional relationships between nodes. It enhances adaptability to flexible user inputs and enables meticulous control over the geometry of objects in the generated scenes. The visual enhancement module enriches the visual fidelity of text-only nodes by constructing visual representations using text embeddings. Furthermore, our relation predictor leverages node representations to infer absent relationships between nodes, resulting in more coherent scene layouts. Extensive experimental results demonstrate that MMGDreamer exhibits superior control of object geometry, achieving state-of-the-art scene generation performance. Project page: https://yangzhifeio.github.io/project/MMGDreamer.