AssetGen: Deployable 3D Asset Generation at Interactive Speed

The evolution of 3D asset creation has been driven by increasing demands in gaming, AR/VR, and industrial design. Early methods relied on manual modeling, which is slow and labor-intensive. Recent advances include neural implicit representations like NeRFs, Gaussian Splatting, and mesh-based generative models, which improve quality but often lack real-time performance. Techniques such as Triplanes, Wavelet features, and primitive-based models have attempted to balance detail and efficiency. Despite these efforts, achieving rapid, high-quality, and deployable assets remains a challenge, especially on resource-constrained devices. The need for automated, fast, and scalable solutions has motivated research into diffusion-based models and GPU-accelerated geometry processing, culminating in systems like AssetGen.

Existing 3D generation methods face a trade-off between quality and speed. High-fidelity models often take minutes to generate, unsuitable for interactive workflows. Many approaches rely on multi-stage pipelines with manual steps, hindering scalability. The challenge is to produce detailed, ready-to-render assets within seconds, compatible with real-time engines and mobile platforms. Achieving this requires innovations in model architecture, inference acceleration, and integrated geometry and texture processing, which have not been fully realized in prior work.

AssetGen introduces several key innovations: 1) a two-stage VecSet diffusion model for dense shape prediction, 2) GPU-based geometry post-processing including mesh simplification, hidden face removal, UV unwrapping, and normal baking, 3) synchronized multi-view texture synthesis with GPU-accelerated backprojection and inpainting, 4) end-to-end system optimization via model distillation, kernel tuning, and pipeline parallelism. These components work synergistically to deliver high-quality assets within strict latency constraints. The system supports two configurations—standard and Flash—tailored for quality and responsiveness, respectively. This holistic design enables rapid, scalable, and deployable 3D asset generation from a single image.

• �� Input a single image, extract features with DINOv2 encoder. • MeshGen encodes dense point clouds via a conditional VAE, then employs DiT diffusion transformer for progressive denoising, generating detailed shape latent codes. • Model distillation reduces diffusion steps from 120 to 30, accelerating inference. • The dense mesh undergoes GPU-based post-processing: hidden face removal, UV unwrapping, and normal baking. • Texture generation synthesizes multi-view images conditioned on the processed mesh, fused into a UV atlas via GPU backprojection and inpainting. • The entire pipeline is optimized through kernel tuning, parallel execution, and asynchronous data transfer, achieving sub-15 seconds latency in Flash mode.

The evaluation uses AssetBench and CharacterBench benchmarks, measuring geometric fidelity and texture quality. On H100 GPUs, the system produces assets in ~30 seconds, with a 1024px texture resolution, comparable to commercial solutions. Human blind tests confirm superior detail and consistency. Ablation studies demonstrate the impact of diffusion step reduction, multi-view synchronization, and GPU-accelerated post-processing. The system performs robustly across diverse object categories and complex geometries, validating its industrial relevance and potential for real-time applications.

AssetGen achieves high-fidelity 3D assets with baked normals and UVs in approximately 30 seconds, outperforming open-source methods in detail preservation and texture coherence. The Flash version reduces latency to 14 seconds, enabling rapid previews. Quantitative metrics show comparable or superior performance in shape accuracy, surface detail, and visual quality. User evaluations favor AssetGen for its realism and consistency, especially in complex scenes. The system maintains performance across various device constraints, demonstrating its scalability and practical deployment potential.

This technology enables instant creation of 3D models for gaming, AR/VR, and industrial prototyping. Developers can generate assets from a single reference image, streamlining workflows and reducing costs. It supports mobile and low-power devices, broadening accessibility. Future integration with interactive editing and multi-view inputs could further enhance creative control, making AssetGen a cornerstone for automated content pipelines in entertainment, manufacturing, and education.

While fast, the system may struggle with highly detailed or cluttered scenes, where fine textures or complex geometries are underrepresented. Dependence on single images limits accuracy in occluded or ambiguous cases. Hardware constraints still impose limits on ultra-low latency scenarios. Future work should focus on multi-view inputs, robustness to input quality, and further model compression to extend applicability to edge devices.