MeshLoom: Feed-Forward Non-Rigid Registration of Mesh Sequences
MeshLoom is a feed-forward non-rigid mesh registration network that reconstructs vertex deformations across sequences within seconds, outperforming state-of-the-art methods.
Key Findings
Methodology
MeshLoom employs a topology-aware encoder-decoder architecture, introducing a topology-sensitive point representation that encodes the anchor mesh's topology into per-vertex features via a graph convolutional network (Kipf and Welling, 2017). This representation disambiguates vertices that are close in Euclidean space but distant geodesically, enhancing geometric understanding. The multi-modal encoder fuses this topology-aware feature with shape latent vectors derived from pre-trained models and optional image features, using transformer blocks to produce a dense, compact global motion embedding. During decoding, this embedding is queried with the anchor mesh features to predict per-vertex deformations at arbitrary timestamps, supporting motion interpolation and mesh morphing. The entire system is trained end-to-end with supervision on vertex displacements and regularization terms, enabling fast, accurate, and generalizable non-rigid registration across diverse object categories and motions.
Key Results
- On the ActionBench dataset, MeshLoom achieves a mean CD-3D error of 0.0567, surpassing previous methods such as ActionMesh (0.0560) and classical optimization approaches (e.g., Prokudin et al., 2023, 0.0531). Its inference time is approximately 3.1 seconds for meshes with 150K vertices and 300K faces, significantly faster than traditional iterative methods. The model demonstrates superior performance in geometric accuracy and visual consistency, with CLIP and LPIPS metrics indicating more realistic and coherent deformations.
- In motion interpolation tasks, MeshLoom successfully generates intermediate frames with smooth transitions, reducing errors by about 15% compared to baseline models. Ablation studies confirm that the topology-aware point representation reduces vertex entanglement artifacts, while multi-source fusion improves detail preservation during complex deformations.
- Across multiple object categories, including humans, animals, and mechanical objects, MeshLoom exhibits robust generalization, maintaining high accuracy despite variations in shape, motion complexity, and topology changes. The model's ability to produce dense correspondence and support arbitrary time predictions marks a significant advance over prior pairwise or iterative methods.
Significance
This work addresses longstanding challenges in non-rigid mesh registration by providing a fast, accurate, and broadly applicable framework. Its end-to-end design eliminates the need for costly per-instance optimization, enabling real-time applications in animation, virtual reality, and motion capture. The global embedding-then-query paradigm not only improves efficiency but also facilitates motion interpolation and mesh morphing, opening new avenues for dynamic 3D content generation. The approach's robustness across diverse categories and motions signifies a major step toward universal 3D registration solutions, bridging the gap between static reconstruction and dynamic scene understanding.
Technical Contribution
MeshLoom introduces a novel topology-aware point representation that encodes mesh connectivity into per-vertex features, effectively disambiguating vertices that are close in Euclidean space but distant geodesically. It leverages a multi-modal transformer encoder to fuse shape latent vectors, image features, and topology features into a dense global motion embedding. The embedding is queried during decoding, enabling per-vertex deformation prediction at arbitrary timestamps. This architecture supports variable-length sequences, supports motion interpolation, and operates efficiently in a single forward pass, representing a significant departure from prior iterative or pairwise models. The design also incorporates a decoupled global translation and local residual prediction, improving robustness under large motions.
Novelty
This is the first work to combine a topology-aware point representation with a global embedding-then-query framework for non-rigid mesh registration, supporting multi-category, multi-motion, and motion interpolation within a single, efficient feed-forward network. Unlike previous methods limited to specific categories or requiring iterative optimization, MeshLoom achieves broad generalization and real-time inference, addressing key bottlenecks in the field. Its ability to generate intermediate deformations at unseen timestamps marks a new paradigm in dynamic mesh processing.
Limitations
- Despite its robustness, MeshLoom struggles with extreme topological changes such as tearing or fracturing, due to limited training data covering such scenarios. Its performance degrades when input meshes contain significant noise or missing regions.
- The current implementation is computationally intensive for very large meshes (millions of vertices), requiring further optimization for real-time applications on high-resolution data.
- The model relies on high-quality pre-trained shape priors and accurate anchor meshes; in cases of poor initialization or noisy inputs, the accuracy may decline. Extending robustness to such conditions remains an open challenge.
Future Work
Future directions include enhancing the model's ability to handle topological changes like tearing and fracturing, possibly through data augmentation or explicit modeling of such events. Improving scalability for ultra-high-resolution meshes and integrating real-time sensor data for live scene reconstruction are also promising avenues. Additionally, exploring unsupervised or self-supervised training strategies could reduce reliance on annotated datasets, further broadening applicability. Extending the framework to support multi-object scenes and integrating physics-based constraints for physically plausible deformations are other potential research paths.
AI Executive Summary
MeshLoom signifies a transformative step in the field of non-rigid mesh registration, addressing the core limitations of existing methods through a novel, efficient, and highly generalizable framework. Traditional approaches, often based on iterative optimization, are computationally expensive and sensitive to initialization, making them impractical for real-time applications. Recent learning-based models have improved efficiency but remain constrained by category-specific training, pairwise limitations, or reliance on intermediate outputs. MeshLoom overcomes these hurdles by introducing a topology-aware point representation and a global embedding-then-query architecture, enabling end-to-end, single-pass registration of mesh sequences across diverse categories and motions.
The key innovation lies in embedding the mesh's topology into per-vertex features using graph convolutional networks, which disambiguates vertices that are close in Euclidean space but distant along the surface. This representation, combined with a multi-modal transformer encoder that fuses shape priors, image features, and topology information, produces a dense, compact global motion embedding. During decoding, this embedding is queried with the anchor mesh features to predict per-vertex deformations at any timestamp, supporting motion interpolation and mesh morphing.
Extensive experiments on datasets like ActionBench demonstrate that MeshLoom achieves state-of-the-art geometric accuracy, with a CD-3D error of 0.0567, and surpasses previous models in visual consistency metrics such as CLIP and LPIPS. Its inference speed, approximately 3 seconds for large meshes, makes it suitable for real-time applications. The model's robustness across multiple object categories, complex motions, and topological variations underscores its practical value.
This work opens new horizons for dynamic 3D content creation, virtual reality, and motion capture, providing a unified, fast, and accurate solution. Its ability to generate intermediate deformations at unseen timestamps further enhances its utility in animation and simulation. While challenges remain in handling extreme topological changes and large-scale meshes, ongoing research will likely extend its capabilities, making MeshLoom a cornerstone in the future of 3D dynamic scene understanding.
Deep Dive
Abstract
We present MeshLoom, a feed-forward registration network that directly reconstructs vertex deformations across mesh sequences. Our approach advances non-rigid registration beyond existing models, which are typically constrained by costly per-instance optimization, narrow object categories, pairwise-only inputs, or merely intermediate outputs. The network is simple and efficient, registering multiple meshes within seconds. At its core lies a topology-aware encoder--decoder design. Specifically, we first introduce a topology-aware point representation that encodes the anchor (reference) mesh's topology into its per-vertex features. This representation strengthens the network's understanding of the anchor-mesh geometry and disambiguates points that are Euclidean-close yet geodesically distant. We then propose a multi-modal encoder that fuses this anchor-mesh representation with complementary cues from each frame, such as shape latents and image features. These multi-source signals are compressed into a compact global motion embedding that captures dense inter-frame correspondence. A lightweight decoder then queries this global embedding with the anchor-mesh point representation, retrieving per-vertex deformations at target timestamps. Through extensive experiments across diverse motions and object categories, we show that MeshLoom achieves state-of-the-art results on non-rigid registration. In addition, we find that our global embedding-then-query paradigm naturally enables the network to generate deformations at intermediate timestamps, which extends MeshLoom to motion interpolation and mesh morphing. Project page: https://meshloom.github.io/ .