ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion
ActionMesh integrates temporal diffusion and autoencoder to rapidly generate topology-consistent animated 3D meshes.
Key Findings
Methodology
ActionMesh employs a two-stage architecture: first, a temporal 3D diffusion model extends existing latent diffusion with a time axis to produce synchronized shape sequences; second, a temporal 3D autoencoder converts these independent shape sequences into deformations of a reference mesh, ensuring topology consistency. The approach leverages pretrained 3D priors, Inflated Attention for cross-frame synchronization, and Masked Diffusion for conditional generation. The model supports diverse inputs like monocular videos, text, or mesh+text prompts, enabling fast, end-to-end animated mesh synthesis without rigging or topology issues.
Key Results
- On Consistent4D and Objaverse benchmarks, ActionMesh outperforms SOTA in geometric accuracy and temporal consistency, reducing Chamfer distances to 0.081 (3D) and 0.148 (4D) on ActionBench, with a 10× speed-up (2 minutes for 16 frames). The generated meshes are rig-free and topology-preserving, facilitating downstream texturing and retargeting.
- Quantitative evaluations show significant improvements over DreamMesh4D and LIM, with better shape fidelity and motion continuity. Ablation studies confirm the importance of Inflated Attention and Masked Diffusion modules in enhancing temporal coherence.
- The model demonstrates versatility across multiple modalities, supporting applications like text-driven animation, motion transfer, and extrapolation, with robust performance in complex dynamic scenes.
Significance
This work addresses the critical bottleneck of slow, optimization-based dynamic 3D reconstruction, offering a fast, scalable, and topology-preserving solution. It bridges the gap between static 3D generation and animated content creation, enabling real-time applications in VR, gaming, and film industries. By supporting multi-modal inputs and ensuring high-quality, consistent outputs, ActionMesh paves the way for democratizing 3D content production, reducing reliance on manual rigging and post-processing, and fostering new creative workflows.
Technical Contribution
The core innovation is the integration of a time-extended 3D diffusion model with a temporal autoencoder, allowing independent shape sequences to be transformed into coherent animations with fixed topology. The Inflated Attention mechanism enhances cross-frame feature interaction, while Masked Diffusion enables partial conditioning on known shapes. This combination results in a highly efficient, end-to-end pipeline that surpasses existing methods in speed and quality, opening new avenues for real-time 3D animation synthesis.
Novelty
This is the first work to incorporate a temporal axis into 3D diffusion models for animated mesh generation, achieving topology consistency without explicit rigging. Unlike prior approaches relying on scene-specific optimization or multi-view consistency, ActionMesh offers a unified, fast, and flexible framework capable of handling diverse inputs and producing production-ready animations, marking a significant step forward in dynamic 3D content synthesis.
Limitations
- Despite its speed, the model may struggle with highly complex or detailed scenes, especially under occlusion or rapid motion, leading to potential detail loss.
- Generalization to unseen object categories or extreme scenarios remains limited, requiring further training or fine-tuning.
- Diffusion sampling is computationally intensive, and real-time performance still demands optimization for practical deployment.
Future Work
Future directions include enhancing model generalization through larger and more diverse datasets, optimizing sampling algorithms for real-time inference, and integrating physical simulation for more realistic motion. Exploring multi-modal training with audio or tactile data could further enrich animation quality. Additionally, extending the framework to support higher-resolution meshes and real-time editing will broaden its industrial applicability.
AI Executive Summary
ActionMesh introduces a novel framework combining temporal diffusion and autoencoding to generate animated 3D meshes with high speed and topological fidelity. Traditional methods for dynamic 3D reconstruction often rely on scene-specific optimization or multi-view consistency, which are computationally expensive and limited in flexibility. In contrast, ActionMesh leverages a two-stage approach: first, a temporal 3D diffusion model extends existing latent diffusion architectures by embedding a time axis, producing synchronized shape sequences from diverse inputs such as monocular videos or text prompts. This stage effectively captures temporal dynamics while maintaining independence across frames, addressing common issues like flickering and inconsistent orientations.
The second stage employs a temporal 3D autoencoder that converts these independent shape sequences into deformations of a reference mesh, ensuring topology consistency throughout the animation. This autoencoder builds upon pretrained VecSet-based VAEs, enhanced with cross-shape attention mechanisms and rotary positional embeddings, to accurately model shape variations over time. The combined architecture allows for fast, end-to-end inference—taking approximately 2 minutes for 16 frames—producing rig-free meshes suitable for downstream tasks like texturing and retargeting.
Experimental results on benchmarks such as Consistent4D and the newly introduced ActionBench demonstrate that ActionMesh surpasses existing state-of-the-art methods in geometric accuracy, temporal coherence, and processing speed. The model's versatility is evidenced by its ability to generate animations from various modalities, including text-to-4D and image+text prompts, as well as its capacity for motion transfer and extrapolation. These advances significantly lower the barrier for high-quality 3D animation creation, promising broad impacts across entertainment, virtual reality, and content production industries.
Despite these strengths, some limitations remain, such as challenges in handling highly detailed or occluded scenes and the computational cost of diffusion sampling. Future work aims to improve generalization, optimize inference speed, and incorporate physical realism, paving the way toward real-time, high-fidelity 3D animation generation at scale.
Deep Dive
Abstract
Generating animated 3D objects is at the heart of many applications, yet most advanced works are typically difficult to apply in practice because of their limited setup, their long runtime, or their limited quality. We introduce ActionMesh, a generative model that predicts production-ready 3D meshes "in action" in a feed-forward manner. Drawing inspiration from early video models, our key insight is to modify existing 3D diffusion models to include a temporal axis, resulting in a framework we dubbed "temporal 3D diffusion". Specifically, we first adapt the 3D diffusion stage to generate a sequence of synchronized latents representing time-varying and independent 3D shapes. Second, we design a temporal 3D autoencoder that translates a sequence of independent shapes into the corresponding deformations of a pre-defined reference shape, allowing us to build an animation. Combining these two components, ActionMesh generates animated 3D meshes from different inputs like a monocular video, a text description, or even a 3D mesh with a text prompt describing its animation. Besides, compared to previous approaches, our method is fast and produces results that are rig-free and topology consistent, hence enabling rapid iteration and seamless applications like texturing and retargeting. We evaluate our model on standard video-to-4D benchmarks (Consistent4D, Objaverse) and report state-of-the-art performances on both geometric accuracy and temporal consistency, demonstrating that our model can deliver animated 3D meshes with unprecedented speed and quality.