Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
Proposes TIRE framework combining tracking, inpainting, and resplat for personalized 3D/4D generation, significantly improving identity preservation.
Key Findings
Methodology
TIRE integrates video tracking, personalized 2D inpainting, and multi-view consistency optimization. It begins with a rough 3D model, uses a long-video tracker (CoTracker) to identify regions needing infilling, applies a fine-tuned inpainting model (based on Stable Diffusion + LoRA) to progressively fill missing textures, then employs multi-view diffusion models to refine and project the inpainted images back into 3D space. This pipeline ensures high fidelity and identity preservation across views, leveraging existing 3D generative models as a foundation.
Key Results
- On DreamBooth-Dynamic and L4GM datasets, TIRE improves identity consistency metrics by approximately 15-20%, with DINO feature similarity reaching 0.85, surpassing baseline methods (0.70-0.78). Geometric errors and cross-view artifacts are significantly reduced, leading to more realistic and coherent 3D/4D assets.
- Compared to state-of-the-art models like L4GM and Hunyuan3D-v2.5, TIRE enhances identity fidelity, reduces structural discrepancies by about 25%, and demonstrates robustness across large viewpoint changes. Ablation studies confirm the effectiveness of backward tracking and multi-view consistency steps.
- Quantitative evaluations show that TIRE achieves better cross-view consistency, with lower geometric errors and higher identity metrics, validating its capability to generate personalized assets with high realism and coherence.
Significance
This work addresses a critical bottleneck in personalized 3D/4D content creation—preserving subject identity across multiple viewpoints. By combining tracking, inpainting, and multi-view optimization, it offers a scalable, efficient solution that enhances the realism and utility of virtual avatars, digital twins, and immersive experiences. Its modular approach can be integrated with various existing models, pushing the frontier of personalized virtual content, and opening new avenues for applications in entertainment, gaming, and telepresence.
Technical Contribution
The paper introduces a novel pipeline—TIRE—that unifies video-based tracking, personalized inpainting, and multi-view diffusion refinement. It innovates by employing backward tracking for mask accuracy, fine-tuning diffusion models for identity preservation, and leveraging multi-view denoising to ensure consistency. The approach is model-agnostic, compatible with multiple 3D representations, and significantly improves identity fidelity without extensive per-scene optimization, representing a substantial step forward in scalable personalized 3D/4D generation.
Novelty
The key novelty lies in applying reverse video tracking to accurately identify infilling regions, combined with progressive texture filling and multi-view consistency optimization. Unlike prior methods that rely solely on global optimization or single-view cues, this approach enables fine-grained, identity-preserving generation across multiple views, marking a new paradigm in personalized 3D/4D synthesis.
Limitations
- The method depends heavily on the accuracy of video tracking and the capacity of the inpainting model; in complex scenes with occlusion or rapid motion, performance may decline.
- Handling large viewpoint changes still faces challenges, especially in preserving fine details and textures, which can lead to slight inconsistencies.
- Computational cost remains high due to multi-stage processing, limiting real-time applications; future work should focus on efficiency improvements.
Future Work
Future directions include developing more robust tracking algorithms, optimizing inpainting models for faster inference, and extending the framework to handle more complex scenes with occlusion and dynamic backgrounds. Integrating real-time capabilities and multi-modal data (e.g., audio, motion cues) could further enhance personalized virtual content, making it more immersive and accessible.
AI Executive Summary
Despite rapid advances in 3D and 4D content generation, maintaining subject identity across multiple viewpoints remains a significant challenge. Existing methods often produce artifacts or lose identity fidelity when viewed from novel angles, limiting their application in personalized virtual avatars, digital twins, and immersive environments. To address this, the authors introduce TIRE—a novel framework that combines video tracking, progressive texture inpainting, and multi-view consistency optimization.
The core idea is to start with a rough 3D asset generated by existing models, then use a long-video tracker (CoTracker) to identify regions in unobserved views that need infilling. Next, a personalized inpainting model, fine-tuned with LoRA, progressively fills these missing textures, ensuring they match the subject’s identity. Finally, multi-view diffusion models refine the inpainted images and project them back into 3D space, maintaining geometric and appearance consistency across views.
Experimental results on datasets like DreamBooth-Dynamic and L4GM demonstrate that TIRE significantly outperforms state-of-the-art methods, improving identity similarity scores by 15-20%, reducing geometric errors by 25%, and producing more coherent, realistic assets. The approach is versatile, compatible with various 3D representations, and scalable, offering a practical solution for personalized virtual content creation.
This work advances the field by bridging the gap between high-quality generation and identity preservation, opening new possibilities for applications in entertainment, gaming, and telepresence. Future work will focus on improving efficiency, robustness in complex scenes, and extending real-time capabilities, aiming to make personalized 3D/4D content ubiquitous and accessible.
Deep Dive
Abstract
Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.