PlanCraft: Sketch, Refine, and Furnish for Architect-Inspired Progressive 3D Residential Scene Generation
PlanCraft integrates progressive sketches and spatial constraints, reducing FID by 61.1% in residential scene generation.
Key Findings
Methodology
The approach combines three modules: SketchPlan replays architect drawing processes on 80K real plans to generate multi-level incomplete sketches; PlanCraft-Diff employs a coarse-to-fine diffusion model trained in two stages to refine sketches into precise floor plans; PlanCraft-Agent uses verified room boundaries, rule-based placement, and collision detection to furnish 3D scenes. The system leverages graph structures, vectorization, and diffusion mechanisms, ensuring layout validity and geometric accuracy. Experiments show a 61.1% lower FID than top 2D baselines, with expert-rated spatial rationality surpassing benchmarks by 15 points, even with only 25% completion sketches.
Key Results
- Trained on 80K real plans, PlanCraft reduces FID by 61.1% compared to existing 2D methods; achieves 15-point higher spatial rationality in expert evaluations; even with 25% sketches, outperforms fully specified baselines, validating progressive sketching.
- Recreates architect’s drawing process via SketchPlan, providing multi-level training data; employs two-stage diffusion training for geometric precision; integrates spatial constraints for layout validity, resulting in robust 3D scene assembly.
- Ablation studies confirm the full system outperforms single-stage models by 3.4×, demonstrating the importance of multi-module collaboration.
Significance
This work addresses the long-standing challenge of integrating design progression with layout validity in automated residential scene generation. By mimicking architect workflows and embedding spatial priors, it bridges the gap between 2D layout creation and 3D scene furnishing. The system’s ability to generate plausible, non-overlapping, geometrically valid layouts from sparse sketches paves the way for smarter, more realistic design automation in architecture, interior design, and virtual reality, significantly reducing manual effort and increasing design fidelity.
Technical Contribution
The paper introduces a unified framework combining progressive sketch reconstruction, diffusion-based refinement, and constraint-based scene assembly. It innovates by re-creating architect drawing processes for training, employing a two-stage diffusion model for global-to-local refinement, and integrating rule-based collision-aware placement within verified room boundaries. These advances enable high-quality, geometrically valid layout generation from incomplete inputs, setting new standards in automated design systems.
Novelty
This is the first system to model residential layout generation from incomplete sketches, mimicking real-world design workflows. It uniquely combines progressive sketch reconstruction with diffusion refinement and explicit spatial constraints, bridging the gap between 2D layout and 3D scene generation. Unlike prior work that relies on fully specified inputs or purely generative models, this approach emphasizes incremental design and geometric validity, marking a significant innovation in the field.
Limitations
- The system struggles with highly complex or unconventional layouts, mainly due to limited training data diversity and the challenge of modeling intricate spatial relationships.
- High computational costs for high-fidelity 3D scene generation limit real-time applications; optimization is needed.
- Adaptability to diverse architectural styles and cultural preferences remains limited; future work should incorporate broader datasets and more flexible spatial reasoning.
Future Work
Future directions include integrating multimodal inputs such as voice or gesture commands, enhancing real-time interactivity, and extending multi-scale spatial reasoning to support complex multi-story buildings. Improving computational efficiency and cultural adaptability will further broaden practical deployment, making the system more versatile for diverse architectural needs.
AI Executive Summary
PlanCraft revolutionizes residential scene generation by emulating the natural, progressive workflow of architects. Traditional methods demand fully detailed inputs, often leading to invalid or unrealistic layouts. In contrast, this system introduces a novel pipeline that begins with partial sketches, akin to initial rough drawings, and iteratively refines them into geometrically precise floor plans. The core innovation lies in the integration of SketchPlan, which replays architect drawing processes on 80,000 real plans, providing rich training signals at multiple completeness levels.
Building upon this, PlanCraft-Diff employs a two-stage diffusion model that balances global plausibility with local geometric accuracy. It gradually sharpens incomplete sketches into valid layouts, leveraging a coarse-to-fine strategy. Once a reliable 2D floor plan is established, PlanCraft-Agent takes over, using verified room boundaries and rule-based algorithms to furnish the scene with furniture and interior elements, ensuring non-overlapping, realistic arrangements.
Experimental results demonstrate the system’s superiority: a 61.1% reduction in FID compared to leading 2D methods, and a 15-point increase in expert-rated spatial rationality over existing 3D scene generators. Remarkably, even with only 25% of the sketch completed, the system already outperforms fully specified baselines, highlighting its efficiency and robustness.
This approach not only advances the state of the art in automated interior design but also aligns closely with actual architectural workflows, making it highly practical. Its ability to generate plausible, detailed, and customizable 3D scenes from minimal input opens new horizons for virtual reality, real estate, and personalized design tools. Future work aims to incorporate multimodal inputs, optimize computational costs, and support more diverse architectural styles, further pushing the boundaries of intelligent scene synthesis.
Deep Dive
Plain Language Accessible to non-experts
想象你在画一幅房子的图纸，起初只用几条简单的线条和块块表示大概的布局。随着时间推移，你会不断调整这些线条，逐渐把房间的轮廓、门窗都画得更清楚。传统的电脑程序就像一个笨手笨脚的画家，必须提前告诉它所有细节，才能开始画。而PlanCraft就像一个聪明的朋友，它可以从你随意画的简单线条开始，逐步帮你完善房子的布局，直到变成完整的3D模型。它还能确保房间不重叠、比例合理，就像用尺子和橡皮反复调整一样。这样一来，设计变得更自然、更贴近真实，也更容易实现个性化定制。你只需画出一些粗略的线条，系统就能帮你完成一个漂亮的房子，不用担心细节问题，既省时又省力。
ELI14 Explained like you're 14
你知道吗？设计房子其实就像玩拼图游戏。一开始，你只拼出一些大块块，像是房子的轮廓。然后，你会慢慢把门、窗户、家具都放进去，直到看起来像个完整的房子。以前的电脑程序就像个笨手笨脚的拼图高手，必须提前告诉它所有块块的形状和位置，才能拼好。而PlanCraft就像一个聪明的朋友，它可以从你随意画的简单线条开始，帮你一步步完善房子的布局，直到变成一个完整的3D房屋模型。它还能确保每个房间都不重叠、比例合适，就像用尺子和橡皮反复调整一样。这让设计变得更自然、更贴近真实，也更方便你个性化定制。就像和朋友一起玩拼图一样，有趣又高效！
Abstract
Two structural insights have been overlooked in automated residential floor plan generation. First, design is inherently progressive. Architects begin with rough strokes and refine them over time, whereas existing methods typically require their conditioning representation to be fully specified before generation, a fundamental mismatch with how design actually works. Second, the 2D floor plan is not an optional intermediate but an irreplaceable spatial contract. Once room boundaries, doors, and windows are fixed, furnishing reduces from open-ended spatial reasoning to bounded constraint satisfaction. Bypassing this contract, as existing 3D systems do by delegating layout to language models, yields overlapping rooms and implausible proportions; directly calling general-purpose language models likewise produces geometrically invalid layouts. Guided by these insights, we present PlanCraft. SketchPlan supplies the missing training signal by replaying the architect's drawing process on 80K real floor plans, producing partial sketches at every completeness level. PlanCraft-Diff progressively sharpens an incomplete sketch into a geometrically precise, vectorizable floor plan through a coarse-to-fine strategy. With the spatial contract established, PlanCraft-Agent then furnishes the scene within well-defined room boundaries. Experiments show that PlanCraft achieves a 61.1\% lower FID than the best existing 2D method and surpasses existing 3D systems by 15 points in expert-rated spatial rationality, with a sketch at only 25\% completion already outperforming all fully specified baselines.