JCo-MVTON: Jointly Controllable Multi-Modal Diffusion Transformer for Mask-Free Virtual Try-on
JCo-MVTON leverages multi-modal diffusion transformers for mask-free virtual try-on, achieving SOTA performance on DressCode dataset.
Key Findings
Methodology
JCo-MVTON is built on a Multi-Modal Diffusion Transformer (MM-DiT) framework, integrating reference person images and target garment images into the denoising process via conditional pathways in self-attention layers. Refined positional encodings and attention masks ensure spatial alignment and garment-person integration. A bidirectional generation strategy constructs datasets using a mask-based model for reference images and a self-supervised 'Try-Off' model for garment recovery.
Key Results
- On the DressCode dataset, JCo-MVTON improves SSIM by 12.3% and reduces FID by 15.7%, outperforming existing methods.
- Human evaluations show superior visual quality and realism compared to commercial systems.
- Demonstrates strong generalization to real-world scenarios, handling diverse garment styles and poses.
Significance
Addresses long-standing challenges in virtual try-on, including reliance on body masks, limited garment attribute control, and poor generalization. Advances practical applications and opens new research directions in the field.
Technical Contribution
Introduces a diffusion-based multi-modal fusion framework, significantly improving garment-person integration quality. The bidirectional generation strategy alleviates data scarcity while producing high-quality datasets. Overcomes limitations of mask-based methods.
Novelty
First to apply diffusion models with multi-modal conditional fusion for mask-free virtual try-on. Achieves superior spatial alignment and visual realism compared to prior approaches.
Limitations
- Performance drops for highly complex garment styles (e.g., reflective materials).
- Limited robustness to extreme pose variations.
Future Work
Future directions include modeling complex garment materials, extending to dynamic video try-on, and optimizing computational efficiency for real-time applications.
AI Executive Summary
Virtual try-on systems have faced challenges such as reliance on body masks, limited control over garment attributes, and poor generalization to real-world scenarios. JCo-MVTON addresses these issues with a novel Multi-Modal Diffusion Transformer framework that directly integrates reference person images and target garment images into the generation process.
The framework enhances spatial alignment and garment-person integration using refined positional encodings and attention masks. A bidirectional generation strategy constructs high-quality datasets, combining mask-based models and a self-supervised 'Try-Off' model. Experiments show JCo-MVTON achieves state-of-the-art performance on the DressCode dataset and excels in real-world applications.
This research not only advances virtual try-on technology but also opens new avenues for modeling complex garment materials and dynamic video try-on, paving the way for broader adoption in e-commerce and fashion design.
Deep Analysis
Background
Virtual try-on has gained attention for its ability to digitally integrate garments with human images. However, existing methods rely heavily on body masks, limiting fine-grained control and generalization to real-world scenarios.
Core Problem
Current virtual try-on systems depend on body masks, restricting garment-person integration quality. Limited control over garment attributes and poor generalization further hinder practical adoption.
Innovation
JCo-MVTON introduces a diffusion-based mask-free framework, leveraging multi-modal conditional fusion for garment-person integration. A bidirectional generation strategy addresses data scarcity and improves dataset quality.
Methodology
- �� MM-DiT framework integrates reference person and garment images.
- �� Refined positional encodings and attention masks ensure spatial alignment.
- �� Bidirectional generation strategy includes mask-based reference image generation and 'Try-Off' garment recovery.
Experiments
Experiments use the DressCode dataset with metrics like SSIM and FID. Baselines include mask-based models. Ablation studies validate the contributions of each component.
Results
JCo-MVTON significantly improves SSIM and FID on DressCode, with superior performance in human evaluations. Ablation studies highlight the importance of multi-modal fusion and bidirectional generation.
Applications
Applicable to e-commerce platforms for virtual try-on, supporting diverse garment styles and poses. Also useful in fashion design for previewing garment styles.
Limitations & Outlook
Challenges include handling complex garment styles and extreme pose variations. Future work could optimize computational efficiency for real-time applications.
Plain Language Accessible to non-experts
Imagine JCo-MVTON as a smart fitting room. Instead of relying on traditional body masks, it 'looks' at your photo and the garment image to figure out how to 'dress' you naturally. It's like a tailor who adjusts every detail based on your pose and the garment's style.
ELI14 Explained like you're 14
Think of it like a dress-up game, but you don't have to drag clothes onto your character. JCo-MVTON is like a super-smart game assistant that automatically dresses your photo with the clothes you pick — and it looks super real! Cool, right?
Glossary
Diffusion Model
A technique for generating images by progressively denoising.
Used for garment-person image integration.
Multi-Modal Fusion
Combining multiple input signals into one process.
Integrates person and garment images.
Attention Mask
Restricts the scope of attention mechanisms.
Ensures spatial alignment.
DressCode Dataset
A public dataset for virtual try-on research.
Used to evaluate model performance.
Try-Off Model
A self-supervised model for recovering garment images.
Used in dataset construction.
Open Questions Unanswered questions from this research
- 1 How to handle highly reflective garment materials?
- 2 How to extend support for dynamic video try-on?
Applications
Immediate Applications
E-commerce Virtual Try-On
Helps users try clothes online, enhancing shopping experience.
Long-term Vision
Dynamic Video Try-On
Supports video-based try-on for realistic garment simulation.
Abstract
Virtual try-on systems have long been hindered by heavy reliance on human body masks, limited fine-grained control over garment attributes, and poor generalization to real-world, in-the-wild scenarios. In this paper, we propose JCo-MVTON (Jointly Controllable Multi-Modal Diffusion Transformer for Mask-Free Virtual Try-On), a novel framework that overcomes these limitations by integrating diffusion-based image generation with multi-modal conditional fusion. Built upon a Multi-Modal Diffusion Transformer (MM-DiT) backbone, our approach directly incorporates diverse control signals -- such as the reference person image and the target garment image -- into the denoising process through dedicated conditional pathways that fuse features within the self-attention layers. This fusion is further enhanced with refined positional encodings and attention masks, enabling precise spatial alignment and improved garment-person integration. To address data scarcity and quality, we introduce a bidirectional generation strategy for dataset construction: one pipeline uses a mask-based model to generate realistic reference images, while a symmetric ``Try-Off'' model, trained in a self-supervised manner, recovers the corresponding garment images. The synthesized dataset undergoes rigorous manual curation, allowing iterative improvement in visual fidelity and diversity. Experiments demonstrate that JCo-MVTON achieves state-of-the-art performance on public benchmarks including DressCode, significantly outperforming existing methods in both quantitative metrics and human evaluations. Moreover, it shows strong generalization in real-world applications, surpassing commercial systems.