Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with nonconform
Implement conformal anomaly detection in Python using nonconform to provide statistically calibrated p-values.
Key Findings
Methodology
The paper introduces a conformal anomaly detection method implemented in Python using the nonconform package. This method converts anomaly scores into calibrated p-values for statistical interpretability. The nonconform package integrates with scikit-learn, PyOD, and supports various conformalization strategies, such as split-conformal calibration and data-efficient extensions.
Key Results
- On the UCI Statlog (Shuttle) dataset, using the Isolation Forest model, nonconform achieved an empirical FDR of 0.18 and a statistical power of 0.99.
- By employing split-conformal calibration and cross-validation methods, data efficiency and p-value resolution were significantly improved.
- Experimental results demonstrate that the nonconform method provides reliable anomaly detection capabilities across various scenarios.
Significance
This research addresses the arbitrariness of threshold selection in traditional methods by providing a statistically calibrated anomaly detection approach. By transforming anomaly detection into a statistical testing problem, nonconform enhances the reproducibility and reliability of anomaly detection decisions in both experimental and production settings.
Technical Contribution
Technical contributions include the development of the nonconform Python package, which supports multiple conformalization strategies, provides a mechanism for converting anomaly scores to p-values, and implements false discovery rate control. This method seamlessly integrates with existing anomaly detection models, offering new engineering possibilities.
Novelty
This is the first implementation of conformal anomaly detection in Python, providing a conversion from anomaly scores to calibrated p-values. Compared to traditional methods, it offers statistical interpretability and error control capabilities.
Limitations
- In small sample cases, split-conformal calibration may lead to low p-value resolution, affecting detection capability.
- The assumption of data exchangeability may not apply to all application scenarios.
Future Work
Future work could explore conformal anomaly detection methods in non-exchangeable data environments and optimize the computational efficiency of existing methods on large-scale datasets.
AI Executive Summary
Most anomaly detection systems output scores rather than calibrated decisions, leaving practitioners without clear statistical interpretation when choosing thresholds. This paper introduces a conformal anomaly detection method that addresses this issue by converting anomaly scores into calibrated p-values. The nonconform package, a Python implementation, supports various conformalization strategies and integrates with scikit-learn, providing a unified interface for calibration, p-value generation, and false discovery rate control.
Empirical results show that the nonconform method achieved an empirical FDR of 0.18 and a statistical power of 0.99 on the UCI Statlog (Shuttle) dataset. The package's design makes core conformal anomaly detection workflows more accessible and reproducible in both experimental and production settings.
However, the nonconform method may face challenges with low p-value resolution in small sample cases. Future improvements could involve more efficient data utilization strategies and extensions for non-exchangeable data environments to address these limitations.
Deep Analysis
Background
Anomaly detection is crucial for identifying unusual patterns in data. Traditional methods like Isolation Forest and One-Class SVM rely on score ranking but lack statistical interpretation. Recently, conformal inference methods have emerged, converting scores into calibrated p-values, offering a new solution.
Core Problem
A core issue with traditional anomaly detection methods is the lack of statistical interpretation, leading to arbitrary threshold selection. This not only affects detection reliability but can also result in high false alarm rates.
Innovation
The core innovation of the nonconform package is providing a mechanism to convert anomaly scores into calibrated p-values, supporting various conformalization strategies like split-conformal calibration and cross-validation, significantly improving data efficiency.
Methodology
- �� Implement conformal anomaly detection using the nonconform package.
- �� Convert anomaly scores into calibrated p-values.
- �� Support multiple conformalization strategies, such as split-conformal calibration.
- �� Integrate with libraries like scikit-learn and PyOD.
Experiments
Experiments were conducted using the UCI Statlog (Shuttle) dataset, based on the Isolation Forest model. Methods like split-conformal calibration and cross-validation were employed to evaluate the detection capability and error control effectiveness of nonconform.
Results
In experiments, nonconform achieved an empirical FDR of 0.18 and a statistical power of 0.99, significantly enhancing the reliability and accuracy of anomaly detection.
Applications
Nonconform is suitable for anomaly detection scenarios requiring high reliability and statistical interpretation, such as financial fraud detection and cybersecurity monitoring.
Limitations & Outlook
In small sample cases, split-conformal calibration may lead to low p-value resolution. The assumption of data exchangeability may not apply to all application scenarios.
Plain Language Accessible to non-experts
Imagine you're in a factory where the task is to produce normal products, but occasionally defects appear. Traditional methods are like workers inspecting products by eye, scoring each product without a clear standard for deciding defects. The nonconform method is like introducing a standardized inspection tool that can provide a calibrated score based on product characteristics, helping workers more accurately identify defects.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to find the bad items hidden among a bunch of normal ones. Traditional methods are like using your eyes to look and guess which items seem off. The nonconform method is like giving you a super magnifying glass that can precisely measure each item's features and tell you which ones might be bad. This way, you can find your target faster and more accurately!
Glossary
Conformal Anomaly Detection
A method that converts anomaly scores into calibrated p-values, providing statistical interpretability.
Used to transform anomaly detection from a ranking task into a statistical testing problem.
p-value
In statistical testing, it measures how much the observed result deviates from the null hypothesis.
Used to determine if a new observation is anomalous.
False Discovery Rate
The proportion of false positives among all observations flagged as anomalies.
Used to control the false alarm rate in anomaly detection.
Exchangeability
The assumption that data is statistically symmetric, meaning the order of observations does not affect the result.
A key assumption for conformal inference methods.
Split-Conformal Calibration
A calibration method that divides data into training and calibration sets.
Used to generate calibrated p-values.
Open Questions Unanswered questions from this research
- 1 How to implement conformal anomaly detection in non-exchangeable data environments?
- 2 How to improve p-value resolution in small sample cases?
Applications
Immediate Applications
Financial Fraud Detection
Improve detection accuracy of anomalous transactions using calibrated p-values, reducing false alarm rates.
Long-term Vision
Large-Scale Data Monitoring
Implement efficient anomaly detection in large-scale data environments, supporting real-time monitoring and decision-making.
Abstract
Most anomaly detection systems output scores rather than calibrated decisions, leaving practitioners to choose thresholds heuristically and without clear statistical interpretation. Conformal anomaly detection addresses this limitation by converting anomaly scores into calibrated p-values that are valid under the statistical assumption of data exchangeability, with a growing literature extending this idea beyond that setting. We present nonconform, a Python package for applying conformal anomaly detection within existing machine-learning workflows, and use it as the basis for an implementation-grounded introduction to the field. The package integrates with scikit-learn, PyOD, and custom anomaly detectors, and provides a unified interface for calibration, p-value generation, and false discovery rate control. It supports several conformalization strategies, ranging from simple split-conformal calibration to more data-efficient and shift-aware extensions. Through a progression from foundational concepts to advanced conformalization strategies, complemented by code examples, the paper connects the statistical ideas behind conformal anomaly detection to their practical use in nonconform. Empirical results demonstrate that the implemented methods enable statistically principled anomaly detection. Together, the package and exposition aim to make core conformal anomaly detection workflows more accessible and reproducible in experimental and production-oriented settings.