Do We Train on Test Data? Purging CIFAR of Near-Duplicates
Using CNN nearest-neighbor retrieval, the authors build ciFAIR and show CIFAR accuracy is inflated by roughly 9–14% relatively.
Key Findings
Methodology
The authors train a lightweight CNN on each CIFAR training set and extract L2-normalized vectors from its global-average-pooling layer. For every test image, they retrieve the nearest training image by Euclidean distance; normalization makes this ranking equivalent to cosine distance. Candidate pairs are manually labeled as Exact Duplicate, Near-Duplicate, Very Similar, or Different. Identified test duplicates are replaced with same-domain samples selected from Tiny Images, producing ciFAIR-10 and ciFAIR-100.
Key Results
- CIFAR-10 contains 286 test images duplicated in training and 39 duplicates within the test set, totaling 3.25% of test images. CIFAR-100 contains 891 train-test duplicates and 104 test-test duplicates, totaling 10%. CIFAR-10 has no exact duplicates, whereas CIFAR-100 does, demonstrating that pixel-level filtering is insufficient.
- All six evaluated CNNs lose accuracy on ciFAIR. ResNet-110 error rises from 5.26% to 5.77% on CIFAR-10 and from 26.05% to 29.24% on CIFAR-100. PyramidNet-272-200 shows the largest reported relative error increase on CIFAR-100: 13.67%.
- On duplicated subsets, ResNet-110 obtains 0% error on CIFAR-10 and 2.9% on CIFAR-100, versus 5.3% and 26.1% on the full original tests. This sharp contrast confirms that even near-duplicates are unusually easy to solve through memorization.
Significance
The paper exposes hidden train-test contamination in two canonical benchmarks, showing that reported generalization partly reflects recognition of reused scenes. This matters because modern model differences can be only one or two percentage points, smaller than the bias introduced by duplicated samples. ciFAIR offers a reproducible, same-domain alternative for researchers, benchmark maintainers, and practitioners estimating deployment risk.
Technical Contribution
The contribution is an auditable pipeline combining domain-adapted CNN retrieval with human semantic verification, rather than relying on exact hashes or an arbitrary distance threshold. The four-way annotation scheme captures progressively weaker visual relationships. Replacement images are sampled from Tiny Images, the original CIFAR source, and screened against nearest neighbors to limit domain shift. The authors release duplicate lists, ciFAIR datasets, pretrained models, and a leaderboard.
Novelty
Unlike work restricted to exact pixel duplicates or work that samples an entirely new CIFAR-10 test set, this study systematically audits both CIFAR-10 and CIFAR-100, distinguishes three meaningful similarity levels, and replaces only contaminated images. Its main novelty is methodological and infrastructural: it reframes benchmark integrity as a first-class scientific issue and supplies a usable duplicate-free protocol.
Limitations
- Candidate discovery depends on one lightweight CNN trained on the relevant dataset, while annotation uses a single human and stops after 20 consecutive Different pairs. Borderline semantic matches may therefore be missed, and subjective labeling bias cannot be estimated.
- Replacement selection is manual and CIFAR-100 lacks published labeling instructions. Although candidates come from Tiny Images and are screened for similarity, subtle class or distribution shifts may remain.
- Only six CNN architectures are evaluated; conclusions do not directly cover Vision Transformers, self-supervised pretraining, or modern large-scale training.
Future Work
Future audits should combine multiple encoders, annotators, agreement statistics, perceptual hashing, and self-supervised representations. The same procedure should be applied to ImageNet, fine-grained datasets, and multimodal corpora. Benchmark reports should disclose provenance, contamination rates, confidence intervals, and clean-test results, while studying how capacity and augmentation affect memorization.
AI Executive Summary
CIFAR-10 and CIFAR-100 have served for years as compact laboratories for image-classification progress. Yet their test sets are not entirely novel. Barz and Denzler show that images, or lightly transformed versions of the same scenes, can occur in both training and testing. Exact pixel matching misses changes in contrast, translation, stretching, and color; augmentation can make such variants especially easy for a network to recognize.
The authors train a lightweight CNN, extract L2-normalized global-average-pooling features, retrieve nearest training neighbors, and manually classify candidate pairs as Exact Duplicate, Near-Duplicate, Very Similar, or Different. They identify 286 train-test duplicates in CIFAR-10 and 891 in CIFAR-100, corresponding to 3.25% and 10% of test images. Replacement samples are selected from Tiny Images, the common source of both datasets, to create ciFAIR-10 and ciFAIR-100.
Across Plain-11, ResNet-110, WRN-28-10, DenseNet-BC, ResNeXt-29, and PyramidNet-272-200, error increases on the cleaned tests. ResNet-110 rises from 26.05% to 29.24% error on CIFAR-100, a 12.25% relative increase; the largest reported relative increase is 13.67%. Rankings remain largely stable, suggesting contamination did not completely reorder architectural progress. The broader lesson is sharper: conventional CIFAR scores include a memorization dividend, while ciFAIR provides a more credible estimate of performance on genuinely unseen images.
Deep Analysis
Background
CIFAR-10 and CIFAR-100 each contain 50,000 training and 10,000 test images, with 10 and 100 classes respectively. Both derive from Tiny Images, an approximately 80-million-image web collection resized to 32×32 pixels. Architectures such as ResNet-110, Wide ResNet, DenseNet, ResNeXt, and PyramidNet have progressively improved benchmark accuracy. Because gains are often only 1–2 points, even modest overlap between splits can materially distort scientific comparisons.
Core Problem
A test set should represent unseen data. Web collection and random splitting can place the same scene, crop, or post-processed variant in different partitions. Contrast shifts, translations, scaling, and color changes evade exact deduplication, while data augmentation makes these variants easy to match. The resulting score conflates abstraction with memorization.
Innovation
The paper contributes four elements. First, it uses learned, domain-adapted CNN features rather than pixels alone. Second, manual four-way annotation reduces false positives from simple thresholding. Third, it replaces only contaminated test images with candidates from Tiny Images, preserving the remaining benchmark. Fourth, it releases ciFAIR, duplicate lists, pretrained models, and a verifiable leaderboard.
Methodology
- �� Train a lightweight CNN on the relevant CIFAR training set.
- �� Extract L2-normalized vectors from global average pooling; retrieve each test image’s nearest training neighbor using Euclidean distance, equivalent to cosine ranking.
- �� Sort pairs by distance and inspect the test image, neighbor, distance, and pixel-difference visualization in a GUI.
- �� Assign Exact Duplicate, Near-Duplicate, Very Similar, or Different; also inspect duplicates within the test set.
- �� Select Tiny Images replacements and reject candidates resembling any of three retrieved neighbors.
- �� Retrain six published CNN systems using their original code and training recipes, then compare error on CIFAR and ciFAIR.
Experiments
The study evaluates Plain-11, ResNet-110, WRN-28-10, DenseNet-BC(L=190,k=40), ResNeXt-29(8×64d), and PyramidNet-272-200 on CIFAR-10/100. Metrics are absolute error, percentage-point gap, and relative gap. Since suitable pretrained models were unavailable, the authors retrained models with reported schedules, optimizers, and regularization. Duplicate-subset versus full-test performance provides the main diagnostic comparison; no conventional parameter ablation is reported.
Results
Train-test duplicates number 286 for CIFAR-10 and 891 for CIFAR-100; test-test duplicates add 39 and 104. Average error increases by 0.41 points on ciFAIR-10 and 2.73 points on ciFAIR-100. PyramidNet’s CIFAR-100 error rises from 17.05% to 19.38%, a 13.67% relative increase. Rankings remain stable on CIFAR-100; only WRN and DenseNet exchange positions on CIFAR-10.
Applications
Researchers should report ciFAIR results alongside conventional CIFAR scores when claiming architectural or training improvements. Dataset curators can use domain-trained feature retrieval followed by human review to audit image collections. Industrial teams should validate models on provenance-independent, duplicate-free data, especially when comparing high-capacity systems whose memorization may mask weak real-world generalization.
Limitations & Outlook
The process is labor-intensive, relies on one encoder and annotator, and may miss semantic duplicates outside the learned feature neighborhood. Manual Tiny Images replacement can introduce subtle distribution or labeling differences, particularly because CIFAR-100 instructions were not publicly specified. The CNN-only evaluation does not establish effects for ViTs or self-supervised models. Future work needs multi-annotator audits, automated perceptual deduplication, and broader benchmark coverage.
Plain Language Accessible to non-experts
Imagine a school that gives students a practice booklet and then an exam. The exam is supposed to contain unfamiliar questions. But some questions are copied from the booklet, while others are the same questions with different colors, spacing, or numbers. A student may score highly simply by recognizing them, not by understanding the lesson.
That is what the authors investigate in CIFAR. They first make a visual “filing system” that puts pictures looking alike next to one another. People then inspect the closest pairs: some are the same picture, some show the same scene after editing, and some are merely very similar. Suspicious exam pictures are replaced with new pictures from the same large image collection, creating ciFAIR.
The result is striking: about 3.25% of CIFAR-10 test pictures and 10% of CIFAR-100 test pictures have a counterpart in training. When those pictures are replaced, recognition becomes harder. The lesson is not that earlier models were useless; rather, their scores included points earned from familiar examples. A fair exam should measure what students can do with new questions.
ELI14 Explained like you're 14
Think of a photo-guessing game. You practice with thousands of pictures, then face a final round that should be totally new. But someone secretly puts a practice picture into the final round after changing its colors or stretching it. You might shout the answer because you remember the picture—not because you understand what is in it!
This paper checks whether CIFAR has that problem. The researchers use a small CNN to find training pictures that look closest to each test picture. Humans inspect the matches and sort them into exact copies, edited copies, very similar pictures, or different ones. Then they fetch replacement pictures from Tiny Images and build ciFAIR-10 and ciFAIR-100.
They find matches for 3.25% of CIFAR-10 tests and 10% of CIFAR-100 tests. ResNet-110 makes 26.05% errors on original CIFAR-100 but 29.24% on ciFAIR-100. So the cleaned exam is harder because it removes easy memory clues.
Does that mean every old leaderboard is fake? Not quite! The model ranking mostly stays similar, so stronger models usually remain stronger. The important message is to test fairly: if we want to know whether AI really learned a visual idea, we must show it pictures it has never effectively seen before!
Glossary
Near-duplicate
Images depicting the same camera scene but altered by color, translation, scaling, or other processing. It is harder to detect than a pixel-identical copy but can leak test information.
The paper’s principal contamination category.
CNN feature space
A vector space in which a convolutional network represents images; nearby vectors generally indicate visual similarity. Here it is derived from global average pooling.
Used for nearest-neighbor mining.
L2 normalization
Dividing a vector by its Euclidean norm so that its length equals one. Euclidean and cosine similarity then induce the same ranking.
Applied before retrieval.
Data augmentation
Creating transformed training examples through operations such as crops, shifts, and color changes. It can improve recognition of near-duplicate variants.
Explains why contamination is easy to exploit.
ciFAIR
CIFAR test sets in which identified duplicates are replaced by screened samples from Tiny Images. The original training sets remain unchanged.
The released benchmark resource.
Generalization
Performance on genuinely unseen examples rather than memorized training items or their close variants. It is the intended target of held-out evaluation.
The capability the paper seeks to measure.
Open Questions Unanswered questions from this research
- 1 How strongly do model capacity, augmentation, self-supervised pretraining, and architecture type affect near-duplicate memorization? The study tests only six CNNs, so modern systems remain uncharacterized.
- 2 How complete and reproducible is the manual audit? Multiple annotators, agreement scores, stronger representations, and automated perceptual methods are needed to estimate missed contamination.
Applications
Immediate Applications
Fair model benchmarking
Research groups can report ciFAIR-10/100 error alongside standard CIFAR results and release model weights. This separates architectural gains from easy recognition of reused scenes and makes comparisons more reproducible.
Dataset contamination audits
Curators can train an in-domain CNN, retrieve cross-split nearest neighbors, and manually review candidates. The workflow is useful for web-scale classification and fine-grained datasets where pixel hashes miss edited copies.
Long-term Vision
Auditable benchmark ecosystem
Future benchmarks could publish provenance, duplicate graphs, versioned clean tests, uncertainty estimates, and verifiable model files. Such infrastructure would make leaderboard scores more predictive of deployment on novel data.
Abstract
The CIFAR-10 and CIFAR-100 datasets are two of the most heavily benchmarked datasets in computer vision and are often used to evaluate novel methods and model architectures in the field of deep learning. However, we find that 3.3% and 10% of the images from the test sets of these datasets have duplicates in the training set. These duplicates are easily recognizable by memorization and may, hence, bias the comparison of image recognition techniques regarding their generalization capability. To eliminate this bias, we provide the "fair CIFAR" (ciFAIR) dataset, where we replaced all duplicates in the test sets with new images sampled from the same domain. We then re-evaluate the classification performance of various popular state-of-the-art CNN architectures on these new test sets to investigate whether recent research has overfitted to memorizing data instead of learning abstract concepts. We find a significant drop in classification accuracy of between 9% and 14% relative to the original performance on the duplicate-free test set. The ciFAIR dataset and pre-trained models are available at https://cvjena.github.io/cifair/, where we also maintain a leaderboard.