Meshy T2: Fast Native Mesh Generation with Flow Matching
Meshy T2 uses flow matching for fast native mesh generation, completing image-to-mesh conversion in 6 seconds with leading geometric fidelity.
Key Findings
Methodology
Meshy T2 employs a flow-matching-based framework, including a vertex-set mesh VAE and a two-stage flow matching model. It first uses image-conditioned voxel flow to sketch a coarse occupancy scaffold, then mesh flow fills vertex latent tokens, enabling high-precision geometry and artist-style topology.
Key Results
- Experiments show Meshy T2 achieves state-of-the-art geometric fidelity, completing image-to-mesh generation in a median of 6 seconds, over 10x faster than autoregressive methods.
- Vertex budget effectively controls face count, and generated meshes excel in multi-part asset support, with components emerging directly from connectivity.
- Ablation studies demonstrate flow matching models balance generation speed and precision effectively.
Significance
This study addresses the slow inference and error accumulation issues of traditional autoregressive mesh generation methods, significantly improving interactive 3D asset creation efficiency. It has major implications for industries like gaming, film, and VR, advancing practical high-quality mesh generation.
Technical Contribution
Meshy T2 introduces a flow-matching-based mesh generation framework, achieving single-pass decoding of vertex latent tokens without quantization or welding issues, and supporting multi-part asset generation. This design surpasses existing methods in speed and geometric precision.
Novelty
Meshy T2 innovates by integrating flow matching models for mesh generation, combining image-conditioned voxel flow and mesh flow to enable efficient coarse-to-fine generation, distinctly different from autoregressive methods.
Limitations
- The model is sensitive to input image quality; low-resolution images may lead to missing mesh details.
- It cannot directly handle dynamic scene mesh generation, requiring further extension.
- Support for complex topology generation remains limited.
Future Work
Future research could explore dynamic scene mesh generation, support for complex topologies, and further optimization of flow matching models for efficiency and precision.
AI Executive Summary
Meshy T2 is a flow-matching-based fast mesh generation framework designed to address the slow inference and error accumulation issues of traditional autoregressive methods. At its core is a vertex-set mesh VAE that decodes vertices, edge connectivity, and face winding order in a single pass, preserving high-precision geometry and artist-style topology.
The generation process involves two stages: image-conditioned voxel flow creates a coarse occupancy scaffold, and mesh flow fills vertex latent tokens, supporting multi-part asset generation. Experiments show the method achieves state-of-the-art geometric fidelity, completing image-to-mesh generation in a median of 6 seconds, over 10x faster than existing methods.
Meshy T2 has significant implications for interactive 3D asset creation in industries like gaming, film, and VR. Despite limitations such as sensitivity to input quality and challenges with complex topologies, the framework opens promising directions for future research and applications.
Deep Analysis
Background
Mesh generation is a core technology in modern 3D pipelines, widely used in gaming, film, and VR. Traditional methods often rely on autoregressive sequence generation, which suffers from slow inference and error accumulation, limiting interactive applications.
Core Problem
Current mesh generation methods face bottlenecks in inference speed, error accumulation, and support for complex topologies, making them unsuitable for real-time interactive asset creation.
Innovation
Key innovations of Meshy T2 include:
- �� Flow matching models combining image-conditioned voxel flow and mesh flow for coarse-to-fine mesh generation.
- �� Vertex-set mesh VAE enabling single-pass decoding without quantization or welding issues.
- �� Multi-part asset generation with components emerging directly from connectivity.
Methodology
Meshy T2's generation process includes:
- �� Image-conditioned voxel flow: sketches a coarse occupancy scaffold to provide a shape framework.
- �� Mesh flow: fills vertex latent tokens, decoding vertices, edge connectivity, and face winding order.
- �� Vertex budget control: allows users to control face count effectively.
Experiments
Experiments use benchmark datasets like ShapeNet and ABC. Compared to autoregressive methods, Meshy T2 excels in geometric fidelity and generation speed. Ablation studies validate the critical role of flow matching models.
Results
Meshy T2 achieves state-of-the-art geometric fidelity, completing image-to-mesh generation in a median of 6 seconds, over 10x faster than traditional methods. Vertex budget effectively controls face count and supports multi-part asset generation.
Applications
Meshy T2 is ideal for gaming, film, and VR applications requiring high-quality meshes and real-time generation.
Limitations & Outlook
The model is sensitive to input image quality, and low-resolution images may lead to missing details. Additionally, support for complex topologies needs further optimization.
Plain Language Accessible to non-experts
Imagine building a LEGO model. Traditional methods are like assembling each piece step-by-step, which is slow and error-prone. Meshy T2 acts like a smart assistant that sketches the overall structure first and then quickly fills in the details to complete the LEGO model.
ELI14 Explained like you're 14
Meshy T2 is like a super-fast LEGO-building robot! It looks at a picture, sketches a rough frame, and then magically fills in the details to create a complete model. It's 10x faster than older methods — cool, right?
Glossary
Flow Matching
A deep learning technique for generating continuous latent vectors.
Used in the coarse-to-fine mesh generation process.
Vertex-Set Mesh VAE
A variational autoencoder enabling single-pass decoding of vertices and connectivity.
Core component of Meshy T2.
Voxel Flow
A model that generates a coarse occupancy scaffold.
Provides the overall shape framework.
Mesh Flow
A model that fills vertex latent tokens.
Generates details and connectivity.
Vertex Budget Control
Controls mesh complexity by setting vertex count.
Enables effective face count management.
Open Questions Unanswered questions from this research
- 1 How can dynamic scene mesh generation be achieved?
- 2 Complex topology generation remains an open challenge.
- 3 How to optimize performance for low-resolution input images?
Applications
Immediate Applications
Game Asset Generation
Supports real-time creation of high-quality game models, enhancing development efficiency.
Film Effects Production
Rapidly generates mesh assets for complex scenes, saving production time.
Long-term Vision
VR Creation Tools
Develop real-time interactive mesh generation tools to advance VR content creation.
Abstract
Polygonal meshes are the standard surface representation of modern 3D pipelines, and generating high-quality meshes with artist-style topology is essential for film, gaming, and interactive 3D applications. Mainstream approaches serialize a mesh into a token sequence and decode it autoregressively, which is slow at inference and sensitive to error accumulation, making them impractical for interactive asset creation. We present Meshy T2, a fast native mesh generation framework built on flow matching. At its core is a vertex-set mesh VAE that encodes a mesh into one continuous latent token per vertex and decodes vertices, edge connectivity, and face winding order in a single pass, preserving high-precision geometry and artist-authored topology without vertex quantization or welding. Generation proceeds as a coarse-to-fine cascade of two flow-matching models: an image-conditioned voxel flow first sketches the overall shape as a coarse occupancy scaffold, and a mesh flow then populates the scaffold with per-vertex latent tokens, conditioned on the image, the scaffold, and a requested vertex budget. This design delivers three practical capabilities: interactive generation speed through parallel flow-based synthesis; effective face-count control through the requested vertex budget; and native support for multi-part assets, whose components emerge directly from the generated connectivity. In our experiments, Meshy T2 achieves state-of-the-art geometric fidelity and completes end-to-end image-to-mesh generation within a median of 6 seconds, over an order of magnitude faster than autoregressive baselines. Code and weights will be available at https://github.com/meshy-dev/meshy-t2.