BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization
BrickAnything uses geometry-conditioned autoregressive modeling with structure-aware tokenization to generate stable, geometrically faithful brick structures, reducing rollback by 70%.
Key Findings
Methodology
This paper introduces BrickAnything, a point cloud-based, geometry-conditioned autoregressive framework that employs structure-aware tree tokenization to encode local attachment relations in brick assemblies. The model uses a pre-trained Michelangelo encoder to extract geometric features from input point clouds, which serve as conditions for a transformer-based sequence generator (OPT-350M). The structure-aware tokenization transforms the brick assembly into a sequence of local attachment tokens, explicitly modeling parent-child relations. During training, a reward-guided Direct Preference Optimization (DPO) aligns the model with buildability objectives, combining geometric fidelity (voxel IoU, Chamfer Distance) and structural stability scores. During inference, validity constraints filter invalid placements, and a stability-guided rollback mechanism corrects unstable structures by tracing back to parent bricks, significantly reducing invalid states and improving physical realism.
Key Results
- On ShapeNet and Objaverse datasets, BrickAnything achieved a 83.4% stable structure rate, IoU of 0.586, and an average rollback count of 0.422, outperforming baseline methods such as BrickGPT-style tokenization and heuristic search. The model’s structure-aware tokenization reduced rollback by 80%, and DPO training improved geometric and stability metrics by 4-6%. Ablation studies confirmed the importance of each component, with removal of structure-aware tokenization or rollback leading to performance drops of over 50%.
- Compared to heuristic methods like Legolization, which failed on complex shapes, BrickAnything reliably produced physically feasible and geometrically accurate structures. The incorporation of local attachment relations and buildability rewards resulted in more robust and realistic assemblies, especially in challenging cases where prior methods struggled. Quantitative metrics showed consistent improvements across all test subsets.
- Ablation experiments demonstrated that the full model’s combination of structure-aware tokenization, validity constraints, and rollback reduced invalid placements by 70%, increased stability by 20%, and enhanced geometric fidelity, confirming the synergy of the proposed components.
Significance
This work advances the field of 3D structure generation by integrating explicit geometric and structural modeling, bridging the gap between digital geometry and physical assembly. It addresses key challenges in ensuring physical stability and geometric accuracy simultaneously, enabling applications in automated construction, virtual prototyping, and robotic assembly. The structure-aware tokenization provides a new way to encode local dependencies, improving the interpretability and robustness of sequence-based generation. The framework’s ability to incorporate buildability rewards and stability correction mechanisms marks a significant step toward practical, reliable automated design systems, with potential impacts on architecture, manufacturing, and digital twin technologies.
Technical Contribution
The core technical innovations include the structure-aware tree tokenization scheme, which encodes local attachment relations explicitly, and the integration of geometry-conditioned sequence generation with a buildability-aware reward via DPO. The model combines point cloud encoding with a transformer decoder, enabling multi-modal, geometry-guided generation. The validity-constrained decoding ensures local correctness, while the stability-guided rollback enhances global physical feasibility. These contributions collectively enable more accurate, stable, and physically realizable brick structure synthesis, surpassing prior methods that lacked explicit structural modeling or relied solely on heuristic optimization.
Novelty
This is the first work to incorporate structure-aware tree tokenization into autoregressive brick generation, explicitly modeling local attachment relations rather than relying on global spatial sequences. The combination of point cloud geometry encoding, structure-dependent sequence prediction, and buildability-aware reward optimization represents a novel integration that significantly improves physical stability and geometric fidelity. Unlike previous approaches limited to semantic or 2D cues, this method directly leverages 3D geometric information and local structural dependencies, setting a new standard for data-driven brick structure synthesis.
Limitations
- Despite improvements, the model may still struggle with highly irregular or extremely complex geometries where point cloud encoding and local attachment assumptions break down. The reliance on high-quality training data limits generalization to unseen or noisy inputs.
- The computational cost of training and inference remains high, especially due to the iterative rollback process and reward computation, which could hinder real-time applications.
- While the stability-guided rollback improves robustness, it cannot guarantee global stability in all cases, especially under extreme load or dynamic conditions. Further integration with physics simulation could enhance reliability.
Future Work
Future directions include integrating multi-modal inputs such as images and textual descriptions to enhance flexibility, developing more efficient encoding schemes for complex geometries, and incorporating physics simulation during training to improve global stability. Extending the framework to support dynamic or deformable structures, as well as real-world robotic assembly, are promising avenues. Additionally, exploring reinforcement learning-based strategies for end-to-end stability optimization could further improve robustness and efficiency.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在搭建一个用积木拼成的城堡。每一块积木都必须正确连接，才能让城堡既漂亮又稳固。以前的方法就像你自己慢慢拼，容易拼错或城堡倒塌。而这项新技术就像有个聪明的机器人助手，它可以根据你的设计图，自动帮你挑选合适的积木，按照正确的顺序拼装。它不仅知道每块积木的形状和位置，还能判断城堡是否会倒。这个机器人学会了很多城堡的搭建技巧，能在保证稳固的同时，还能快速完成拼装。未来，它还能帮建筑师设计更复杂的房子，甚至帮机器人自己建房子！
ELI14 Explained like you're 14
想象你在玩乐高积木，你要搭一个又高又稳的城堡。以前的方法就像你自己慢慢拼，容易出错或倒塌。而现在，有个超级聪明的机器人助手，它能看你的设计图，帮你挑选合适的砖块，然后一步步拼好，确保城堡不会倒。它知道每块砖该怎么连接，能在发现某块不稳时，自动重新调整。这个机器人学会了很多搭建技巧，能帮你更快、更稳地建出漂亮的城堡。将来，它还能帮建筑师设计更复杂的建筑，甚至帮机器人自己建房子！
Abstract
Generating physically buildable brick structures from 3D shapes requires more than geometric reconstruction: the output must also satisfy discrete part constraints and structural stability. Existing brick generation methods either rely on heuristic optimization, which can break down when the target 3D shape does not admit a feasible structure under predefined constraints, or generate brick sequences without explicitly modeling the underlying 3D geometry and assembly relations. In this work, we present BrickAnything, a geometry-conditioned autoregressive framework for generating buildable brick structures from diverse 3D representations. BrickAnything uses point clouds as a unified geometric interface and predicts brick sequences that reconstruct the target shape under assembly constraints. To model structural dependencies among bricks, we introduce a structure-aware tree tokenization, which represents brick structures through local attachment relations. This formulation makes sequence generation more consistent with the physical construction process, and reduces invalid intermediate states. We further introduce preference-based alignment post-training, validity-constrained decoding and adaptive rollback to improve buildability objectives such as stability and geometric fidelity. Extensive experiments demonstrate that BrickAnything produces geometrically faithful and physically realizable brick structures, and that the proposed tokenization effectively reduces rollback and regeneration compared with conventional ordering strategies.