Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection
Study finds PA replacement metrics are robust at N=1 but inflate at N>1; recommends PR metrics.
Key Findings
Methodology
The study uses an independent adversarial stress-test to evaluate the robustness of post-point-adjustment metrics against no-skill detectors. Experiments were conducted on six benchmark datasets to analyze the performance of different metrics under single-run and multi-run scenarios, revealing metric inflation behavior with varying N values.
Key Results
- At N=1, no replacement metric is gameable, with random detectors reaching 90% of the best real detector's score on at most 11% of series.
- At N=3, affiliation-F1 becomes gameable on 25% of series, reaching 0.98 at N=41.
- PR metrics and PA%K remain stable across all N values, close to anomaly prevalence.
Significance
The study highlights the inflation issue of post-point-adjustment metrics under multi-run scenarios, emphasizing the importance of disclosing N values when reporting metrics and recommending PR metrics to resist inflation. This has significant implications for metric selection and reporting methods in time-series anomaly detection.
Technical Contribution
This work provides the first independent validation of post-point-adjustment metrics' robustness against no-skill detectors, introduces a pip-installable stress-test tool, and reveals the behavior differences between ROC and PR metrics under extreme class imbalance.
Novelty
This study is the first to independently evaluate the robustness of post-point-adjustment metrics from an adversarial perspective, particularly under multi-run scenarios, filling a gap in existing literature regarding independent validation of these replacement metrics.
Limitations
- Experiments are primarily conducted on the UCR dataset, and performance on other datasets may vary.
- The study does not consider the impact of complex detectors, focusing mainly on no-skill detectors.
Future Work
Future research could explore validating these metrics' robustness on more datasets and developing new metrics to reduce inflation under multi-run scenarios.
AI Executive Summary
Time-series anomaly detection is crucial in fields like web services, spacecraft telemetry, industrial control, and finance. However, traditional point-adjustment metrics have been criticized for being overly permissive to random anomaly scores. To address this, researchers proposed a suite of replacement metrics, including PA%K, range-based precision/recall, affiliation precision/recall, and VUS-ROC/PR. Yet, whether these replacements can resist no-skill detectors under multi-run scenarios remains an unresolved question.
This study conducts an independent adversarial stress-test to evaluate these replacement metrics across six benchmark datasets. Results show that under a single run (N=1), all replacement metrics are robust against manipulation. However, under multi-run scenarios, affiliation-F1 and ROC metrics exhibit significant inflation, especially at N=41, where affiliation-F1 becomes gameable on 98% of series.
The study recommends disclosing N values when reporting metrics and prioritizing PR metrics, as they demonstrate stronger robustness under multi-run scenarios. This finding provides important guidance for metric selection and reporting methods in the field of time-series anomaly detection.
Deep Analysis
Background
Time-series anomaly detection plays a vital role in various fields, including web services, spacecraft telemetry, industrial control, and finance. Traditional point-adjustment protocols have been criticized for their permissiveness to random anomaly scores, prompting researchers to develop a suite of replacement metrics designed to enhance robustness and prevent no-skill detectors from achieving high scores.
Core Problem
The core problem is whether these replacement metrics can resist manipulation by no-skill detectors under multi-run scenarios. Due to the common practice of seed shopping in machine learning, metrics may exhibit inflation under multi-run scenarios, allowing no-skill detectors to achieve scores similar to real detectors.
Innovation
The core innovation of this study lies in independently validating the robustness of post-point-adjustment metrics against no-skill detectors, particularly under multi-run scenarios. By comparing the performance of ROC and PR metrics, the study reveals the behavioral differences of AUC under extreme class imbalance.
Methodology
- �� Conduct experiments on six benchmark datasets, including UCR, SMD, SMAP, MSL, NAB, and PSM.
- �� Evaluate 12 metrics under single-run and multi-run scenarios.
- �� Use a pip-installable stress-test tool for experiments.
- �� Analyze metric inflation behavior across different N values.
Experiments
The experimental design includes using six benchmark datasets to evaluate the performance of 12 metrics under single-run and multi-run scenarios. Random detectors and real detectors are used for comparison, analyzing metric inflation behavior across different N values.
Results
Results show that under a single run (N=1), all replacement metrics are robust against manipulation. However, under multi-run scenarios, affiliation-F1 and ROC metrics exhibit significant inflation, especially at N=41, where affiliation-F1 becomes gameable on 98% of series.
Applications
The study's findings have significant implications for metric selection and reporting methods in time-series anomaly detection. It recommends disclosing N values when reporting metrics and prioritizing PR metrics.
Limitations & Outlook
Experiments are primarily conducted on the UCR dataset, and performance on other datasets may vary. Additionally, the study does not consider the impact of complex detectors, focusing mainly on no-skill detectors. Future research could explore validating these metrics' robustness on more datasets.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and point adjustment is like assuming the entire pot of soup is cooked just because one piece of meat is done. But that's not accurate, as other parts might still be raw. To better assess the soup's doneness, you decide to taste small portions and record each taste. This way, you get a more accurate picture of the whole pot's status. This method is similar to the replacement metrics mentioned in the paper, which assess anomalies more precisely rather than relying on a single point's status.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to find hidden treasures on a map. Before, finding just one treasure point would give you all the rewards. But that's too easy! So now, the game rules have changed, and you need to find more treasure points to get the rewards. This is like the new methods in the paper, which evaluate anomaly detectors more strictly rather than just looking at one point's performance. This makes the game more challenging and fairer!
Glossary
Point Adjustment
An evaluation method for anomaly detectors where detecting any point within an anomaly segment counts as detecting the entire segment.
Used as a traditional evaluation standard for anomaly detection.
ROC Curve
Receiver Operating Characteristic curve, used to evaluate a detector's classification performance.
Used to compare the robustness of different metrics.
PR Curve
Precision-Recall curve, used to evaluate a detector's performance on imbalanced datasets.
Analyzed for PR metrics' behavior under extreme class imbalance.
VUS (Volume-Under-the-Surface)
A comprehensive metric that evaluates detector performance across multiple dimensions.
Evaluated as one of the replacement metrics.
Seed Shopping
The practice of selecting the best result from multiple runs, which may lead to metric inflation.
Used to analyze metric performance under multi-run scenarios.
Open Questions Unanswered questions from this research
- 1 How can these metrics' robustness be validated on more datasets? Current studies focus mainly on the UCR dataset, and performance on other datasets may vary.
- 2 How can new metrics be developed to reduce inflation under multi-run scenarios? Existing metrics show inflation under multi-run scenarios.
Applications
Immediate Applications
Anomaly Detection Optimization
By using PR metrics, anomaly detection accuracy can be improved, especially under multi-run scenarios.
Metric Reporting Improvement
Disclosing N values when reporting metrics to enhance result transparency and credibility.
Long-term Vision
Robust Metric Development
Develop new metrics to reduce inflation under multi-run scenarios, enhancing overall anomaly detection performance.
Abstract
Point-adjustment (PA), for years the default scoring protocol in time-series anomaly detection (TSAD), was shown by Kim et al. (2022) to award near-perfect F1 to random anomaly scores. The field adopted a suite of replacement metrics (PA%K, range-based precision/recall, affiliation precision/recall, and Volume-Under-the-Surface, VUS, ROC/PR). We ask, independently and adversarially, whether these resist no-skill detectors on real benchmarks, and find the answer turns entirely on one overlooked variable: N, the number of random attempts an adversary reports the best of. Under a single honest run (N=1), not one replacement metric is gameable on any of six benchmarks (UCR, SMD, SMAP, MSL, NAB, PSM): a random detector reaches 90% of the best real detector's score on at most 11% of series for affiliation-F1, 5% for the ROC family, and 2% for the PR-based metrics and PA%K. But under best-of-N reporting, the seed-shopping endemic to ML, the metrics split sharply. affiliation-F1 and every ROC-based metric inflate steeply, affiliation crossing gameable (25% of series) by N=3 and reaching 0.98 at the full pool (N=41), the ROC family crossing by N=9-11; the PR-based metrics and PA%K stay near-flat at every N, floored near the anomaly prevalence (the lone exception is NAB at large N). A paired test finds VUS-ROC inflated on 131 series where its sibling VUS-PR is not, and never the reverse. The ROC-vs-PR split follows from the order-statistic behaviour of AUC under extreme class imbalance (a random PR-AUC is floored at prevalence); affiliation inflates by a second route, its extreme single-run leniency (already fragile at N=1). We release a pip-installable stress-test harness, and recommend reporting single-run scores or disclosing N and preferring PR-based metrics, which resist best-of-N inflation on nearly every benchmark.