Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting
HO-HE post-generation curation balances fidelity and diversity, reaching 95% on CIFAR-10 with 300K synthetic images.
Key Findings
Methodology
The paper introduces HO-HE post-generation curation. MoCo v3 extracts normalized features, and a directed within-class 1-NN graph separates Homogeneous (HO) representative samples from Heterogeneous (HE) non-redundant variation. Synthetic candidates are scored separately against both partitions using fidelity and diversity, then ranked under S_p=αS_p^div+(1−α)S_p^fid, with α=0.5 by default. No generator retraining is required.
Key Results
- On CIFAR-10, the method reaches 95% accuracy with about 300K synthetic images, whereas RandSelect and RealScore need roughly 500K and SBSim needs substantially more. On SVHN, about 100K curated samples match the real-data baseline, while competing methods require more than 200K.
- On ImageNet-1K with EDM2 data, ViT-B/16 fine-tuning reaches 73.14% and 74.76% with 1M and 3M selected images, respectively, versus Random at 71.90% and 73.65%, and SBSim at 72.18% and 74.14%.
- The criterion remains useful after task-tuned generation: on ImageNet-100, 30K curated images reach 79.00% versus 78.14% for a 50K baseline, a 40% reduction; on VOCaug, JoDiffusion mIoU rises from 73.61% to 74.94% at 10K images.
Significance
The work shifts attention from generating more data or modifying a generator to using an existing synthetic pool more intelligently. It directly addresses the structural tendency of modern generators to repeat canonical class modes while missing intra-class variation, reducing generation and tuning costs. Academically, it offers an explicit, computable balance between fidelity and diversity. Industrially, teams can curate third-party or frozen generator outputs without accessing model internals, and can approach real-data performance with fewer training examples.
Technical Contribution
The central technical contribution is a directed 1-NN HO-HE partition rather than a global centroid or k-means split. Proposition 1 shows that HO is the unique inclusion-minimal nearest-neighbor cover under unique neighbors, preserving every sample's nearest-neighbor reconstruction cost. The method then uses the HO centroid as an anchor and matches each HE feature to an HO feature. Directional cosine geometry measures deviation from canonical patterns. Construction costs O(n²d), while batched scoring costs O(|D_S|nd).
Novelty
The novelty is not simply combining similarity and diversity, but defining diversity relative to a structurally identified canonical-versus-nonredundant partition. CLIP-Align, RealScore, and SBSim mainly reward label or image similarity, which can reinforce redundancy. HO-HE explicitly models generator bias toward HO patterns, operates with sample-only access, and can be applied after both generic and task-customized generators.
Limitations
- Curation depends on feature encoder ϕ. If MoCo v3 is poorly matched to the target domain, local neighborhoods, HO/HE membership, and directional diversity scores may be unreliable, especially for fine-grained or medical imagery.
- The method assumes labeled real reference data and a fixed synthetic pool. It cannot recover a semantic mode absent from that pool, nor can it repair systematic label errors or severe visual artifacts by selection alone.
- Large classes make the O(n²d) 1-NN construction expensive, and the paper provides limited evidence for extreme long-tail, web-scale, or highly imbalanced settings.
Future Work
Future work should study domain-adaptive or task-aware encoders, label-free HO-HE discovery, and dynamic budgets for long-tail classes. The score could incorporate uncertainty, task gradients, or active-learning value. Extensions to video, 3D, medical, and multimodal generation would require temporal or cross-modal consistency. Approximate nearest-neighbor indexing, distributed scoring, and streaming curation could reduce the quadratic construction cost at industrial scale.
AI Executive Summary
Generative models can now produce vast numbers of realistic images, yet quantity does not guarantee usefulness. They often reproduce each class’s canonical appearance—standard poses, viewpoints, and backgrounds—while underrepresenting intra-class variation. Existing remedies typically fine-tune the generator, engineer prompts, or apply inference-time guidance, all of which incur cost and require model-specific expertise. This paper asks a more operational question: can a fixed synthetic pool become more valuable through selection alone?
The proposed HO-HE framework uses MoCo v3 features and a directed within-class 1-NN graph to divide real data into Homogeneous (HO) representatives and Heterogeneous (HE) variation. HO provides local canonical coverage; HE contains samples that are not nearest-neighbor representatives for others but preserve non-redundant diversity. Synthetic images are scored separately for fidelity to each partition and for directional deviation from canonical anchors, using S_p=αS_p^div+(1−α)S_p^fid. The method requires no generator access beyond sample output.
Across SVHN, CIFAR-10, Tiny-ImageNet, ImageNet-1K, ImageNet-100, and VOCaug, and with EDM, EDM2, JoDiffusion, ResNet, and ViT models, curation consistently improves data efficiency. On CIFAR-10, 300K selected images reach 95% accuracy, compared with roughly 500K for RandSelect and RealScore. On ImageNet-1K, fine-tuning reaches 73.14%/74.76% with 1M/3M images; ImageNet-100 achieves 79.00% with 30K rather than 50K; VOCaug mIoU rises from 73.61% to 74.94%. The result is a complementary data-centric layer, not a replacement for better generators.
Deep Analysis
Background
Synthetic data has become a scalable response to data scarcity in visual learning, with generators such as EDM and EDM2 producing high-quality samples for CIFAR-10 and ImageNet. However, label noise, real-to-synthetic distribution gaps, and mode collapse remain. CLIP-Align emphasizes image-label agreement, while RealScore and SBSim emphasize image-image similarity. These methods improve fidelity but can repeatedly select canonical images and fail to recover intra-class variation.
Core Problem
Given a labeled real reference set D_R, a fixed synthetic pool D_S, and budget k, the goal is to choose A⊆D_S that improves downstream performance on the real distribution. The difficulty is structural: generators and discriminators preferentially model easy, representative HO patterns, while HE examples are harder to recognize yet carry useful variation. A single similarity ranking therefore amplifies rather than corrects generator bias.
Innovation
The paper contributes four linked ideas. First, a directed 1-NN HO-HE split preserves local geometry better than global centroid partitioning. Second, Proposition 1 formalizes HO as a minimal nearest-neighbor cover. Third, partition-conditioned scoring jointly rewards semantic fidelity and deviation from canonical redundancy. Fourth, the procedure is generator-agnostic and can be plugged into task-customized synthesis without retraining or generator access.
Methodology
- �� Feature extraction: MoCo v3 produces ℓ2-normalized features f_i with cosine distance 1−〈f_i,f_j〉.
- �� Partitioning: each real sample points to its closest same-class neighbor; positive in-degree nodes form HO, while zero in-degree nodes form HE.
- �� Anchors: the HO anchor is centroid C_HO; each HE anchor is its nearest matched HO feature.
- �� Fidelity: S_p^fid=cos(F_syn,F_p), rewarding semantic alignment with partition p.
- �� Diversity: S_p^div=−cos(R_p−F_p,F_syn−F_p), rewarding movement away from canonical directions.
- �� Selection: S_p=αS_p^div+(1−α)S_p^fid; rank separately in HO and HE branches, take Top-k, and merge. Experiments use α=0.5 unless stated otherwise.
Experiments
The main benchmarks are SVHN, CIFAR-10, Tiny-ImageNet, and ImageNet-1K, with EDM or EDM2 synthetic pools. Plug-in tests use ImageNet-100 and VOCaug, including inference-time intervention and JoDiffusion as task-tuned generation baselines. Selection baselines are RandSelect, CLIP-Align, RealScore, and SBSim. Downstream backbones include ResNet-18/50, EfficientNet-B0, and ViT-B/16. Metrics are classification accuracy, ImageNet Top-1 accuracy, segmentation mIoU, and OOD performance across nine related benchmarks.
Results
The method improves data efficiency across scales. On SVHN, about 100K selected images match the real-data baseline, versus more than 200K for alternatives. On CIFAR-10, 300K images reach 95%, while RandSelect and RealScore need about 500K. For ImageNet-1K ViT-B/16, scratch training with 1M/3M images gives 71.49%/74.02%, and fine-tuning gives 73.14%/74.76%. ImageNet-100 improves from a 78.14% 50K baseline to 79.00% with 30K. VOCaug mIoU improves from 73.61%/73.87% to 74.94%/76.04% at 10K/15K images.
Applications
Teams can generate a large image pool once, use a modest labeled real reference set to build HO-HE partitions, and curate data for classification, segmentation, or augmentation. This is particularly useful when a third-party generator cannot be modified. Autonomous driving, industrial inspection, remote sensing, and medical imaging could benefit from retaining rare views or defect patterns, although safety-critical deployment still requires human review and domain validation.
Limitations & Outlook
The approach depends on reference quality, class labels, and encoder-domain compatibility. Approximate or mismatched features can corrupt local neighborhoods and diversity estimates. The O(n²d) graph construction is costly for very large classes, while the method cannot create modes absent from the synthetic pool. It also ranks samples rather than fully modeling downstream loss. Future work should explore adaptive encoders, label-free and long-tail variants, task-gradient objectives, approximate search, and extensions to video, 3D, and multimodal data.
Plain Language Accessible to non-experts
Imagine a factory producing school supplies. It can manufacture millions of notebooks, but it keeps making the same standard notebook: identical cover, size, and layout. The notebooks are attractive and usable, yet the factory rarely makes notebooks for left-handed students, field trips, unusual subjects, or different reading levels. If a school simply buys everything, it receives a huge pile with surprisingly little variety.
HO-HE curation acts like a careful purchasing manager. First, the manager studies a real school’s supplies. HO items are the standard examples that represent many similar items; HE items are less typical but add something that cannot be replaced by another item. The manager then examines factory products using two questions: “Does this look like something the school really needs?” and “Is it genuinely different, or just another copy of the standard item?”
The final order mixes reliable basics with useful variation. It does not ask the factory to rebuild its machines, so an existing inventory can be reused. In the paper, this simple reordering lets CIFAR-10 reach 95% accuracy with about 300,000 selected images instead of roughly 500,000 for some alternatives. The lesson is that a better shopping list can make an existing factory much more useful.
There is an important limit: a manager cannot select a product the factory never made. The manager can also make mistakes if the comparison rules do not fit the school’s needs. Thus, selection complements better production; it does not replace it.
ELI14 Explained like you're 14
Picture a game that automatically creates monster cards. It generates hundreds of thousands of cards, but it loves the same “standard fire dragon”: front-facing, shiny, and standing in one pose. Real players know dragons can be rainy, injured, tiny, huge, viewed from behind, or hiding in a forest. If you train a card-recognition bot only on standard dragons, it may panic when the dragon looks different.
This paper creates a smart card curator. It first studies a real collection and finds two kinds of cards. HO cards are good representatives of common designs. HE cards are the unusual-but-useful ones that add new poses, colors, or situations. Then every generated card gets two scores: one asks whether it looks realistic and belongs to the right monster category; the other asks whether it adds variety instead of copying the standard dragon again.
The curator keeps a balanced collection. On CIFAR-10, about 300,000 chosen images reached 95% accuracy, while some other methods needed around 500,000. On ImageNet-100, the system used 30,000 images and reached 79.00%, beating a 50,000-image baseline at 78.14%. That is like training your bot with fewer cards but teaching it more useful situations!
The trick is not magic: if the generator never creates a sea dragon, the curator cannot discover one. Also, the curator’s idea of “similar” depends on how it looks at images. Still, the paper shows that smarter selection can make a giant generated pile much more valuable.
Glossary
Homogeneous subset (HO)
A set of locally representative real samples that cover common class semantics. Technically, HO consists of nodes with positive in-degree in the directed within-class 1-NN graph.
It models canonical patterns that generators tend to reproduce heavily.
Heterogeneous subset (HE)
A set of real samples containing non-redundant intra-class variation. Technically, HE consists of zero-in-degree nodes in the directed 1-NN graph and should not be interpreted as noise.
It provides the reference for recovering diversity missed by generators.
Fidelity
How well a synthetic image matches the semantics of a real reference partition. In this paper it is measured by cosine similarity in feature space.
It forms S_p^fid and suppresses implausible or misaligned samples.
Diversity
How much a candidate departs from canonical patterns while remaining tied to a real reference. The method uses the negative cosine between reference-to-anchor and reference-to-synthetic directions.
It forms S_p^div and favors non-redundant variation.
EDM / EDM2
Diffusion-based image-generation systems designed for high-quality synthesis. They are used here to create fixed synthetic pools rather than being modified by HO-HE.
EDM supports SVHN, CIFAR-10, and Tiny-ImageNet; EDM2 supports ImageNet-1K.
Post-generation curation
Selecting or filtering samples after generation rather than changing the generator. It can operate with sample access and labels without retraining.
HO-HE is a post-generation curation framework.
Open Questions Unanswered questions from this research
- 1 When the real reference set is small, noisy, or long-tailed, can 1-NN reliably distinguish canonical coverage from meaningful rarity? Robust estimation and uncertainty-aware partitioning are still needed.
- 2 Feature-space geometry is encoder-dependent. Automatic domain-specific encoder selection or task-aware representation learning could substantially affect the reliability of HO, HE, and diversity scores.
- 3 Can the framework handle video, 3D, and multimodal data while preserving temporal consistency, spatial structure, and cross-modal alignment?
Applications
Immediate Applications
Synthetic classification-set compression
A team with a large EDM, EDM2, or internal generation pool can use a modest labeled real set to build HO-HE references, then curate a smaller training set. The expected benefits are lower storage and training costs while maintaining or improving accuracy.
Segmentation augmentation
Autonomous-driving, industrial-inspection, or remote-sensing teams can apply HO-HE after task-tuned generators such as JoDiffusion. After validating masks and labels, selection can preserve both ordinary scenes and rare geometric variations; VOCaug results show measurable mIoU gains.
Long-term Vision
Adaptive long-tail data warehouses
Future data platforms could track representativeness, diversity, uncertainty, and downstream utility continuously, allocating more synthetic budget to undercovered classes. Combined with active learning, generated data could become an optimizable resource rather than a static archive.
Abstract
Recent generative models can produce high-quality synthetic images, offering scalable training training data for data-hungry models. Existing approaches to exploiting this potential typically involve 1) training or fine-tuning generators, or 2) using lightweight post-hoc adaptation like prompt engineering or inference-time guidance, making them generator-specific and expertise-intensive. We study a complementary question: given a fixed pool of generated images, can downstream utility be improved purely by selecting an informative subset? The answer is yes. We show that effective selection must counter a structural bias of modern generators: they tend to over-produce canonical modes of each class while underrepresenting intra-class variation. Building on this insight, we split each real class into a canonical Homogeneous (HO) subset and a non-redundant Heterogeneous (HE) subset, then score synthetic images by a fidelity-diversity criterion that rewards semantic alignment while penalizing canonical redundancy. The method is generator-agnostic and requires no retraining. Across multiple benchmarks, it consistently outperforms state-of-the-art data selection baselines and matches the real-data performance with up to 40% fewer synthetic samples. The same criterion remains effective when applied on top of stronger task-tuned generators, with gains on both classification and segmentation tasks. Post-generation selection is therefore not a substitute for better generators, but a complementary mechanism for improving the utility of synthetic data.