Natural Adversarial Examples
Adversarial filtration created ImageNet-A/O; DenseNet-121 reached only ~2% on A and OOD detection was near chance.
Key Findings
Methodology
The paper introduces adversarial filtration: candidates are collected from iNaturalist, Flickr, DuckDuckGo, or ImageNet-22K, then easy cases are removed using ResNet-50 and 10-crop prediction. Human reviewers retain clear, single-label images that still trigger errors. ImageNet-A contains difficult examples from 200 ImageNet-1K classes; ImageNet-O contains non-ImageNet-1K concepts that are nevertheless assigned known labels with high confidence.
Key Results
- ImageNet-A contains 7,500 images. DenseNet-121 achieves about 2% accuracy, versus usually above 90% on ordinary ImageNet images from the same 200 classes—an approximately 90-point degradation. AlexNet, SqueezeNet, VGG-19, ResNet-50, and other unseen architectures also fail substantially.
- ImageNet-O contains 2,000 OOD images. Using negative maximum softmax probability as the anomaly score, random AUPR is about 16.67%; ResNet-50 obtains about 16.20%, essentially chance-level, showing that unknown concepts are often classified as known ones with high confidence.
- Augmentation gains are modest: the ResNet-50 baseline is 2.17%, adversarial training falls to 1.68%, while AugMix reaches 3.8%, Cutout 4.4%, MoEx 5.5%, Mixup 6.6%, and CutMix 7.3%. ImageNet-21K pretraining raises accuracy to 11.41%.
Significance
The work demonstrates that clean, naturally occurring images can systematically defeat modern vision models without pixel perturbations. ImageNet-A reveals that standard benchmarks may overestimate robustness, while ImageNet-O exposes severe failures under semantic label shift. These datasets turn concerns about shortcut learning and overconfidence into reproducible tests, with direct implications for autonomous driving, medical imaging, robotics, and safety-critical deployment.
Technical Contribution
The technical contribution is the first systematic use of adversarial filtration for image-dataset construction, selecting the hardest rather than merely removing the easiest examples. Fixed ResNet-50 filtering, a second-stage cap of 15 occurrences per confusion, and extensive human review produce diverse, valid failures. ImageNet-O further uses non-ImageNet-1K concepts from ImageNet-22K, keeping collection conditions close to ImageNet while shifting semantics, unlike OOD benchmarks based on noise, textures, or distant scene datasets.
Novelty
Unlike L_p attacks, ImageNet-C corruptions, and ImageNetV2, the paper uses unmodified real-world images and demonstrates transfer across model families. ImageNet-A is a natural adversarial benchmark for classification under input shift; according to the authors, ImageNet-O is the first OOD dataset specifically curated for ImageNet models.
Limitations
- The datasets cover a selected 200-class subset and require months of graduate-student curation. This limits scale, category coverage, and reproducibility compared with automatically generated benchmarks.
- Because ResNet-50 defines the initial filter, the sample distribution may overrepresent its blind spots. Transfer to several CNNs is strong, but coverage of vision transformers and multimodal models is not established.
- ImageNet-O assumes training only on ImageNet-1K; using ImageNet-22K training data would make the task substantially easier.
Future Work
Future work should expand categories and settings, test vision transformers, multimodal models, and video, and automate human-quality control. Promising directions include shape- and causal-feature learning, confidence calibration, uncertainty modeling, and training directly on failure modes. Better ensemble filters may reveal weaknesses not captured by a single ResNet-50.
AI Executive Summary
ImageNet classifiers have achieved impressive benchmark accuracy and are sometimes described as “superhuman.” Yet standard test images are often clear, centered, and rich in familiar contextual cues. The central question of this study is whether models remain reliable when images are natural and recognizable to people but come from rare, complicated real-world configurations.
Hendrycks and colleagues use adversarial filtration: ResNet-50 models apply 10-crop predictions to remove easy candidates, after which researchers enforce confusion diversity and perform human review. The result is ImageNet-A, 7,500 images belonging to 200 ImageNet-1K classes, and ImageNet-O, 2,000 images whose concepts fall outside ImageNet-1K but are confidently misclassified as known classes. FID between ImageNet-A and the ordinary validation or test distributions is about 50.40 and 50.25, respectively, while estimated human accuracy is about 90%.
The failures are striking. DenseNet-121 achieves only about 2% on ImageNet-A, despite ordinary accuracy above 90%; OOD detection reaches roughly 16.20% AUPR, close to the 16.67% random baseline. Adversarial training decreases accuracy to 1.68%, while AugMix, Cutout, MoEx, Mixup, and CutMix reach 3.8%, 4.4%, 5.5%, 6.6%, and 7.3%. ImageNet-21K pretraining improves performance to 11.41%, suggesting that architectural and representational changes—not routine augmentation alone—are needed for robust open-world vision.
Deep Analysis
Background
ImageNet has driven progress in classification, detection, and segmentation, but its test set often contains clean, obvious objects. ImageNetV2 examined benchmark reproducibility, Geirhos et al. showed that classifiers can prefer texture over shape, and ImageNet-C studied synthetic corruptions. This paper targets a different failure source: naturally occurring, unmodified images from long-tail scene configurations that remove easy shortcuts while remaining intelligible to humans.
Core Problem
The study asks whether classifiers can recognize known objects under realistic distribution shift and whether OOD detectors can reject semantically unknown but visually similar inputs. Traditional benchmarks often use Places365, SUN, texture datasets, Gaussian noise, or synthetic blobs, which may be far from ImageNet and therefore make detection artificially easy.
Innovation
- �� First systematic image-dataset use of adversarial filtration.
- �� ImageNet-A selects human-clear but model-hard examples from 200 classes.
- �� ImageNet-O selects non-ImageNet-1K concepts from ImageNet-22K that receive high-confidence known labels.
- �� A 15-occurrence confusion cap increases error diversity.
- �� Clean natural images test cross-architecture transfer beyond L_p threat models.
Methodology
- �� Inputs: weakly labeled images from iNaturalist, Flickr, DuckDuckGo, or ImageNet-22K.
- �� Classification filter: two ResNet-50 systems perform 10-crop inference; ImageNet-A candidates are removed if any crop is correct or if the correct-class confidence exceeds 15%.
- �� OOD filter: negative maximum softmax probability is used as the anomaly score; high-confidence OOD errors are retained.
- �� Diversity control: each confusion is allowed at most 15 appearances.
- �� Human validation: graduate students verify clear, single-class, high-quality images, producing 7,500 ImageNet-A and 2,000 ImageNet-O examples.
Experiments
The study evaluates AlexNet, SqueezeNet, VGG-19, DenseNet-121, and ResNet-50. ImageNet-A uses top-1 accuracy; ImageNet-O uses AUPR, with a random baseline near 16.67%. Fixed-architecture ResNet-50 experiments test adversarial training, Style Transfer, AugMix, Cutout, MoEx, Mixup, CutMix, Places365 pretraining, and ImageNet-21K pretraining.
Results
DenseNet-121 obtains about 2% on ImageNet-A, and multiple unseen architectures show comparable weakness, indicating shared blind spots. The ResNet-50 baseline is 2.17%; CutMix reaches only 7.3%, while adversarial training drops to 1.68%. Places365 pretraining produces 1.56%. ImageNet-21K pretraining raises ImageNet-A accuracy to 11.41% and ImageNet-O AUPR from 16.20% to 21.86%, an improvement that remains far from reliable detection.
Applications
ImageNet-A can serve as a pre-deployment stress test for autonomous vehicles, robots, surveillance, and medical vision. ImageNet-O tests whether systems know when to abstain rather than produce dangerous high-confidence labels. Practical use requires confidence calibration, human escalation, open-set validation, and continuous collection of long-tail failures.
Limitations & Outlook
The benchmark depends on a ResNet-50 filter and costly manual selection, so it may miss failure modes specific to other architectures. Its 200 classes do not represent the full visual world. ImageNet-O also assumes no ImageNet-22K training. Future work should scale to broader categories, video, vision-language systems, and more principled studies of shape bias, causal representations, calibration, and architecture design.
Plain Language Accessible to non-experts
Imagine a warehouse worker trained to recognize products from thousands of catalog photos. In training, shoes usually sit on shelves, apples appear in baskets, and snow shovels appear beside snow. The worker may learn the surroundings rather than the product itself. Move the shoe into mud or place the shovel somewhere unusual, and the worker confidently names the wrong item. The researchers create exactly these situations for computer vision, without editing pixels or adding noise.
ImageNet-A is a difficult exam whose answers are still familiar categories. Humans can identify its images at roughly 90%, but DenseNet-121 gets only about 2%. ImageNet-O is different: it contains unfamiliar products that look somewhat like known ones. A reliable worker should say, “I do not recognize this.” The models instead often give a confident wrong answer; their AUPR is about 16.20%, almost the 16.67% random level.
The team tried several training tricks. Mixing, hiding, or restyling pictures helped a little, with CutMix reaching 7.3%. Much more labeled data from ImageNet-21K raised performance to 11.41%, but the problem remained severe. The lesson is simple: seeing more examples is useful, but the system must also learn what makes an object itself, rather than memorizing scenery and familiar habits.
ELI14 Explained like you're 14
Suppose you play a picture-guessing game. During practice, every cat is photographed indoors, every car is on a road, and every shovel appears beside snow. After a while, you may stop looking at the object and start using the background as a cheat code. Then a cat outdoors or a shovel without snow makes you fail spectacularly. That is what this paper discovers in computer vision!
The researchers collected lots of online images and asked ResNet-50 to find the ones it was most likely to misidentify. Humans checked that the pictures were actually clear and had one main answer. These became ImageNet-A: 7,500 difficult images whose labels the computer had technically learned. Some models normally score above 90%, but DenseNet-121 scores only about 2%. Ouch!
ImageNet-O asks another question: can the computer say “I don’t know”? It shows pictures of concepts outside ImageNet-1K that may look similar to known things. Instead of admitting uncertainty, the model often shouts out a very confident wrong label. Its AUPR is about 16.20%, almost the 16.67% random score. Confidence is clearly not the same as correctness!
The authors test Mixup, CutMix, AugMix, and other tricks. CutMix improves 2.17% to 7.3%, while ImageNet-21K pretraining reaches 11.41%. Better than nothing, but still nowhere near dependable. The big challenge is teaching AI to notice the object itself—not just colors, textures, backgrounds, and habits learned from easy practice pictures.
Glossary
Adversarial filtration
A procedure that removes easy examples and keeps naturally occurring examples that reliably cause model errors. It selects images rather than changing their pixels.
Used to construct ImageNet-A and ImageNet-O.
Natural adversarial example
A real, unmodified input that causes a model to make a mistake. Its difficulty can arise from rare contexts, shortcut cues, or visual ambiguity.
The paper's central object of study.
Out-of-distribution (OOD)
An input whose semantic distribution differs from the data used for training. A robust model should detect it and reduce confidence.
ImageNet-O evaluates OOD detection.
Maximum softmax probability
The largest predicted class probability produced by a classifier. The paper uses its negative as an anomaly score.
Baseline detector for ImageNet-O.
AUPR
Area under the precision-recall curve, summarizing detection quality across thresholds. Higher values indicate better anomaly ranking.
Random ImageNet-O performance is about 16.67%.
Open Questions Unanswered questions from this research
- 1 Why very different CNN architectures share similar natural failure modes remains unclear. A mechanistic theory involving shape bias, causal features, and representation geometry is needed.
- 2 Transfer to vision transformers, multimodal models, and real video has not been systematically established. These systems may share, reduce, or introduce different blind spots.
- 3 Scaling human curation while preserving single-label quality, realistic difficulty, and diverse confusions remains a major dataset-engineering challenge.
Applications
Immediate Applications
Pre-deployment vision stress testing
Teams can evaluate a model on ImageNet-A after ordinary validation to expose reliance on backgrounds, textures, or canonical composition. Very low accuracy should trigger additional long-tail data collection, abstention rules, calibration, and human review before deployment.
Unknown-object detection
Autonomous-driving, robotics, and security systems can use ImageNet-O to test high-confidence recognition failures. In deployment, negative maximum softmax probability should be combined with calibrated thresholds, sensor redundancy, and escalation policies.
Long-term Vision
Open-world robust vision
Future systems should represent object shape, parts, and causal structure rather than merely textures and context, while continuously learning from real failures. This could enable safer open-set recognition and more reliable cross-environment generalization.
Abstract
We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets' real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called ImageNet-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called ImageNet-O, which is the first out-of-distribution detection dataset created for ImageNet models. On ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on ImageNet-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.