Geometry-Aware Dataset Condensation for Diffusion Model Training
Proposes a geometry-aware dataset condensation method to enhance fidelity and distribution coverage in diffusion model training.
Key Findings
Methodology
The paper introduces a geometry-aware distribution alignment method using one-sided partial optimal transport (POT) and statistical regularization for dataset condensation. It includes a geometry-guided greedy selection and swap-based refinement to ensure the selected subset aligns with the full data distribution.
Key Results
- On ImageNet-1K using the DiT-L/2 model, the FID for a 10K image subset decreased from 4.20 (D2C) to 3.43, indicating higher distribution fidelity.
- The method outperforms baseline methods in FID and IS metrics across different data budgets, especially under low data budgets.
- Ablation studies show that the combination of statistical regularization and geometry alignment significantly improves training efficiency and generation quality.
Significance
This research is significant for academia and industry as it addresses the trade-off between dataset size and computational cost in diffusion model training. By employing a geometry-aware selection method, it ensures high-quality generation under limited data budgets, promoting the application of diffusion models in resource-constrained environments.
Technical Contribution
The technical contributions include reformulating the dataset condensation problem as a geometry-aware distribution alignment problem and proposing an efficient two-stage discrete optimization strategy. This method fundamentally differs from existing score-based selection strategies, offering new theoretical guarantees and engineering possibilities.
Novelty
This is the first application of geometry-aware distribution alignment in dataset condensation for diffusion models, differing from traditional difficulty score-based selection methods by providing a finer distribution alignment mechanism.
Limitations
- The method may face insufficient alignment under extremely low data budgets, affecting generation quality.
- Computational cost remains high for high-resolution images.
- Further research is needed on adaptive regularization parameter adjustment for different tasks.
Future Work
Future directions include exploring more efficient alignment algorithms to reduce computational costs and applying the method to multimodal datasets. Additionally, adaptive regularization parameter adjustment for different tasks is a promising area of study.
AI Executive Summary
Dataset condensation aims to construct compact datasets from real data via synthesis or selection. However, existing methods perform poorly in diffusion model training. This paper proposes a geometry-aware distribution alignment method using one-sided partial optimal transport (POT) and statistical regularization for dataset condensation. The method includes a geometry-guided greedy selection and swap-based refinement to ensure the selected subset aligns with the full data distribution.
Experimental results show that the method outperforms existing baselines across different data budgets, especially under low data budgets. On ImageNet-1K using the DiT-L/2 model, the FID for a 10K image subset decreased from 4.20 (D2C) to 3.43, indicating higher distribution fidelity.
This research is significant for academia and industry as it addresses the trade-off between dataset size and computational cost in diffusion model training. Future directions include exploring more efficient alignment algorithms to reduce computational costs and applying the method to multimodal datasets. Additionally, adaptive regularization parameter adjustment for different tasks is a promising area of study.
Deep Analysis
Background
Dataset condensation aims to construct compact datasets to reduce storage and computational costs. However, existing methods perform poorly in diffusion model training as they often fail to preserve the geometric structure required by diffusion models.
Core Problem
Diffusion model training requires high-quality datasets, but large-scale datasets incur high storage and computational costs. Existing synthetic data generation and real subset selection methods fail to effectively preserve the geometric structure of data distributions.
Innovation
The paper proposes a geometry-aware distribution alignment method using one-sided partial optimal transport and statistical regularization for dataset condensation. This method ensures the selected subset aligns with the geometric structure of the full data distribution.
Methodology
- �� Use one-sided partial optimal transport (POT) for geometry alignment.
- �� Ensure distribution fidelity through statistical regularization.
- �� Two-stage optimization strategy: geometry-guided greedy selection and swap-based refinement.
Experiments
Experiments were conducted on ImageNet-1K using DiT-L/2 and SiT-L/2 models, comparing performance across different data budgets. Evaluation metrics include FID and IS, with baselines such as D2C and Herding.
Results
For a 10K image subset, the method's FID decreased from 4.20 (D2C) to 3.43, indicating higher distribution fidelity. Ablation studies show that the combination of statistical regularization and geometry alignment significantly improves training efficiency and generation quality.
Applications
The method is applicable to diffusion model training in resource-constrained environments, enabling high-quality generation under limited data budgets. It has broad applications in image generation and data augmentation.
Limitations & Outlook
The method may face insufficient alignment under extremely low data budgets, affecting generation quality. Computational cost remains high for high-resolution images.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You need to make a delicious dish with limited ingredients. Existing methods are like randomly picking ingredients, which might miss key flavors. This method is like choosing ingredients based on their taste and nutritional value, ensuring each dish retains its original flavor and nutrition. This selection method allows you to make a tasty dish even with limited ingredients.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to build a strong fortress with limited resources. Existing methods are like randomly picking resources, which might lead to a weak fortress. This method is like choosing resources based on their value and use, ensuring each resource is used to its fullest potential. This way, even with limited resources, you can build a strong fortress!
Glossary
Dataset Condensation
The process of constructing compact datasets through synthesis or selection to reduce storage and computational costs.
Used in diffusion model training to reduce data volume.
Diffusion Model
A generative model that produces high-quality samples through iterative denoising.
Requires high-quality datasets for training.
Geometry Alignment
Ensuring the selected subset aligns with the geometric structure of the full data distribution.
Achieved through one-sided partial optimal transport.
Optimal Transport
A mathematical method for finding the optimal matching between two distributions.
Used to achieve geometry alignment in dataset condensation.
Statistical Regularization
Improving model distribution fidelity by aligning feature statistics.
Used to complement geometry alignment to ensure distribution fidelity.
Open Questions Unanswered questions from this research
- 1 How to maintain high-quality generation under extremely low data budgets? Existing methods may face insufficient alignment.
- 2 How does the method perform on multimodal datasets? Further research is needed.
Applications
Immediate Applications
Image Generation
Use the method to train diffusion models in resource-constrained environments for high-quality image generation.
Long-term Vision
Multimodal Data Processing
Apply the method to multimodal datasets to improve data processing efficiency and quality.
Abstract
Dataset condensation aims to construct compact datasets from real data via synthesis or selection. However, existing approaches are ill-suited for diffusion model training: synthetic data generation often yields low-fidelity samples unsuitable for authentic modeling, while real subset selection typically fails to preserve the distributional geometry required by diffusion likelihood objectives. To address this, we propose to reformulate real subset selection as a geometry-aware distribution alignment problem. By incorporating one-sided partial optimal transport, our method selectively aligns a compact subset with the full data distribution while allowing unmatched mass in low-density regions, ensuring the preserved geometric structure necessary for effective diffusion model training. To further ensure distributional fidelity, we complement geometric alignment with lightweight feature-statistics and semantic consistency regularization. An efficient two-stage discrete optimization strategy is proposed to achieve this alignment objective. Extensive experiments across diffusion variants, subset sizes, image resolutions, and training rounds show that our method achieves superior fidelity and distributional coverage in diffusion model training. Codes are available at https://github.com/2018cx/GADC.