MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers
MeshGPT uses decoder-only Transformer with quantized geometric vocabulary to generate compact, sharp 3D meshes, improving shape coverage by 9%.
Key Findings
Methodology
MeshGPT employs a graph convolution encoder to extract local geometric features, which are then quantized into a discrete vocabulary via residual quantization. A ResNet decoder reconstructs triangle vertices from these embeddings. A GPT-style transformer is trained to predict the next token in the sequence, enabling autoregressive mesh generation. The process involves sequence compression to handle long sequences, ensuring efficient training. The model is trained on ShapeNet across 55 categories, optimizing cross-entropy and reconstruction losses, resulting in high-fidelity, sparse meshes.
Key Results
- MeshGPT achieves a 9% increase in shape coverage and a 30-point improvement in FID scores over state-of-the-art methods like PolyGen and GET3D on ShapeNet. Generated meshes exhibit sharp edges, detailed features, and compact topology, closely resembling artist-created meshes. Quantitative metrics confirm superior geometric quality, with lower Chamfer distances and higher coverage. User studies show a strong preference for MeshGPT outputs in both shape and triangulation quality. Ablation studies validate the importance of geometric vocabulary learning, sequence compression, and per-vertex quantization for optimal performance.
- Across categories such as chairs, tables, benches, and lamps, MeshGPT consistently outperforms baselines in shape diversity, visual fidelity, and geometric sharpness. The method effectively balances detail and compactness, generating meshes that are both realistic and computationally efficient. The ability to generate novel shapes beyond training distribution demonstrates its potential for creative applications. The model's robustness is confirmed through extensive ablations, highlighting the critical roles of each component in achieving high-quality results.
Significance
This work advances 3D shape synthesis by directly generating sparse, sharp meshes, overcoming limitations of dense iso-surfaced or point cloud-based methods. It bridges the gap between language-inspired sequence modeling and geometric mesh generation, enabling more controllable, efficient, and artistically faithful 3D content creation. The approach opens new avenues for automated design, virtual reality, and digital fabrication, where high-quality, lightweight meshes are essential. It also sets a foundation for integrating semantic and texture information in future models, pushing the frontier of intelligent 3D content generation.
Technical Contribution
The paper introduces a novel framework combining graph convolutional encoding, residual vector quantization, and autoregressive transformer modeling for mesh synthesis. It innovatively encodes local geometry into a discrete vocabulary, compresses sequences to fit transformer constraints, and employs a decoder-only architecture for efficient, high-fidelity mesh generation. This approach differs from prior methods relying on implicit fields or separate vertex-face models, offering a unified, end-to-end trainable pipeline that captures intricate geometric details while maintaining computational efficiency.
Novelty
This is the first work to adapt large-scale language models for direct triangle mesh generation via a learned geometric vocabulary. Unlike previous methods that generate dense meshes or implicit representations requiring post-processing, MeshGPT models meshes as sequences of discrete tokens, enabling the synthesis of sparse, sharp, and artist-like meshes. Its integration of graph convolutional features, residual quantization, and sequence modeling constitutes a significant innovation, setting a new benchmark in 3D generative modeling.
Limitations
- The model's performance diminishes with extremely complex or high-resolution meshes, where fine details may be oversimplified or artifacts may appear. Its reliance on training data limits generalization to highly novel or out-of-distribution shapes.
- Training requires substantial computational resources, including multiple GPUs over several days, which may hinder widespread adoption.
- Post-processing steps like vertex merging are necessary for clean meshes, adding complexity to the pipeline and preventing fully end-to-end automation.
Future Work
Future directions include integrating texture and material generation, reducing computational costs via model compression, and extending the framework to dynamic or scene-level mesh synthesis. Additionally, exploring multi-modal conditioning (e.g., text, images) could enable more controllable and semantically meaningful shape generation, broadening practical applications in entertainment, design, and virtual environments.
AI Executive Summary
MeshGPT marks a significant leap in 3D shape synthesis by directly generating sparse, detailed triangle meshes through a sequence modeling approach inspired by language models. Traditional methods, such as iso-surfacing from neural fields or point cloud reconstruction, often produce dense, over-tessellated meshes that lack the sharpness and efficiency of artist-crafted models. In contrast, MeshGPT employs a novel pipeline: it first learns a discrete geometric vocabulary via graph convolutional encoding and residual quantization, capturing local topology and geometry with high fidelity. This vocabulary enables representing each mesh as a sequence of tokens, which are then modeled using a decoder-only transformer akin to GPT. The transformer predicts the next token autoregressively, allowing the generation of diverse, compact meshes with sharp edges and intricate details. Extensive experiments on ShapeNet demonstrate that MeshGPT outperforms existing methods like PolyGen, AtlasNet, and GET3D, with a 9% increase in shape coverage and a 30-point reduction in FID scores. User studies further confirm the perceptual superiority of the generated meshes, which closely resemble human-made models in both shape and topology. The approach’s ability to generate novel shapes beyond training data showcases its potential for creative and industrial applications. Despite these advances, challenges remain in handling extremely complex geometries and reducing training costs. Future work aims to incorporate textures, improve efficiency, and extend to scene-level synthesis, promising a new era of intelligent, controllable 3D content creation.
Deep Analysis
Background
三维几何表示在计算机图形学中占据核心地位，早期采用点云、体素等方法。随着深度学习的发展，隐式场和神经场模型逐渐兴起，但生成密集网格仍面临细节不足和后处理复杂的问题。点云方法如PointNet、Diffusion模型虽能生成多样形状，但难以直接用于高质量渲染。Mesh作为表达高效、控制性强的模型，需求不断增长。现有技术多依赖二阶段流程，存在拓扑不一致和细节丢失等难题。本文试图突破这一局限，直接生成稀疏、锐利的网格，提升表达能力。
Core Problem
现有网格生成方法多依赖后续处理，导致密集、过度细化的网格，存储和渲染成本高，难以模拟艺术家手工制作的高效拓扑结构。如何直接生成稀疏、细节丰富且具有锐利边缘的网格，成为关键难题。传统序列模型难以捕获局部几何特征，长序列带来训练难题，缺乏有效的几何词汇表达，限制了生成质量。
Innovation
提出基于大规模语言模型的序列生成框架，将三角形编码为离散的几何词汇，结合图卷积提取局部特征，利用残差量化压缩序列长度。创新点包括：1）几何词汇学习，提升几何一致性；2）序列压缩，适应Transformer长度限制；3）单一解码器结构，简化流程。该方法首次将Transformer应用于网格生成，显著改善了稀疏性和锐利边缘，突破了点云和隐式场的局限。
Methodology
- �� 构建图卷积编码器，输入三角形邻域信息，提取局部几何特征。• 使用残差向量量化，将连续特征离散化为有限的几何词汇。• 通过ResNet解码器，将量化特征还原为三角形顶点坐标。• 训练GPT风格的Transformer，学习序列中下一个词汇的预测，利用交叉熵损失优化。• 生成时，模型自回归采样，预测下一词汇序列，再通过后处理合并顶点，得到最终网格。
Experiments
在ShapeNetV2数据集上训练，涵盖55类，采用形状覆盖率、FID、KID、Chamfer距离等指标评估。对比PolyGen、AtlasNet、GET3D等，进行定量和定性分析。训练参数包括残差深度D=2，序列长度压缩，训练时间约2天，使用多GPU。还进行消融验证，确认几何词汇和序列压缩的关键作用。
Results
MeshGPT在ShapeNet上实现了9%的形状覆盖率提升，FID下降30点，生成的网格锐利、细节丰富，优于所有对比方法。用户偏好调查显示，用户更喜欢MeshGPT的模型，尤其在细节和拓扑一致性方面。消融实验验证了几何词汇和序列压缩的重要性，模型能生成新颖、多样的形状，覆盖训练集和超出训练分布。
Applications
该方法适用于虚拟现实、游戏、工业设计等场景，能自动生成高质量3D模型，减少人工建模时间。未来结合纹理、材质信息，有望实现完整的虚拟场景自动化，为内容创作带来革命性变革。
Limitations & Outlook
模型在极端复杂或高细节场景下仍存在模糊或不连续的问题，训练成本高，依赖大量GPU资源。后处理步骤影响端到端自动化，未来需优化模型结构和推理效率。
Plain Language Accessible to non-experts
想象你在一个工厂里，工人们用不同的工具拼装模型。有些模型需要很多材料和时间，但工人们很熟练，能用少量材料拼出漂亮、锐利的模型。现在，MeshGPT就像一个聪明的机器人，它学会了用一种特殊的“语言”描述每个拼图块（三角形），然后用这台机器人逐个预测下一块，拼出完整的模型。它不像传统方法那样先用很多材料拼出复杂的模型，而是用少量材料，快速拼出细节丰富、边缘锐利的模型，就像工艺师手工雕刻一样。
ELI14 Explained like you're 14
想象你在玩拼图游戏，但不是用普通的拼图块，而是用一种神奇的语言描述每块拼图。这个机器人学会了用这个语言，把每个拼图块变成一个符号，然后用一台超级聪明的电脑（Transformer）逐个猜出下一块符号。它像写故事一样，把所有符号串在一起，最后拼出一个漂亮的3D模型。这个方法让机器人可以快速拼出细节丰富、边缘锐利的模型，比以前的笨重模型更像艺术品。它不仅更快，还能模仿人类手工制作的精细工艺。
Glossary
Transformer（变换器）
一种基于注意力机制的深度学习模型，擅长处理序列数据，能预测序列中下一个元素。论文中用来生成三角形序列。
用于自回归预测网格三角形序列的核心模型。
Residual Quantization（残差量化）
一种将连续特征离散化的技术，通过逐层编码减少误差，提升重建质量。论文中用于几何词汇学习。
将局部几何特征编码成有限的离散词汇。
Graph Convolution（图卷积）
一种在图结构数据上进行特征提取的神经网络操作，捕获邻域信息。用于提取三角形的局部几何特征。
作为编码器的关键部分，提取局部几何信息。
ShapeNet
一个大规模的3D模型数据集，包含各种类别的物体模型，用于训练和评估3D生成模型。
模型在ShapeNet上训练和测试，涵盖55类。
Open Questions Unanswered questions from this research
- 1 如何提升模型在极端复杂或高细节场景中的生成质量，尤其在保持锐利边缘和细节一致性方面，仍是未解难题。
- 2 训练成本较高，未来需优化算法和硬件利用率，以实现更高效的端到端自动化。
- 3 结合纹理和材质信息，使模型生成的内容更真实、更具实用性，也是未来的重要方向。
Applications
Immediate Applications
虚拟内容自动生成
可用于游戏、虚拟现实中的场景和角色模型自动设计，减少手工建模时间，提高内容多样性。
工业设计与原型制作
帮助设计师快速生成符合需求的3D模型原型，加快产品开发流程，提升设计效率。
Long-term Vision
全自动3D内容创作平台
结合多模态信息，实现从文本、图片到完整3D模型的自动生成，推动虚拟世界和数字孪生的发展。
Abstract
We introduce MeshGPT, a new approach for generating triangle meshes that reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields. Inspired by recent advances in powerful large language models, we adopt a sequence-based approach to autoregressively generate triangle meshes as sequences of triangles. We first learn a vocabulary of latent quantized embeddings, using graph convolutions, which inform these embeddings of the local mesh geometry and topology. These embeddings are sequenced and decoded into triangles by a decoder, ensuring that they can effectively reconstruct the mesh. A transformer is then trained on this learned vocabulary to predict the index of the next embedding given previous embeddings. Once trained, our model can be autoregressively sampled to generate new triangle meshes, directly generating compact meshes with sharp edges, more closely imitating the efficient triangulation patterns of human-crafted meshes. MeshGPT demonstrates a notable improvement over state of the art mesh generation methods, with a 9% increase in shape coverage and a 30-point enhancement in FID scores across various categories.