E-Values Expand the Scope of Conformal Prediction
Introduces conformal e-prediction using E-values and Ville’s inequality for flexible, non-exchangeable, sequential coverage guarantees.
Key Findings
Methodology
This paper proposes conformal e-prediction based on E-values, leveraging Ville’s inequality to construct supermartingales that guarantee coverage at any time point. The core algorithm defines nonnegative E-variables (formula 4), builds a supermartingale process, and applies Ville’s inequality for sequential control. The approach accommodates non-exchangeable data, distribution shifts, and ambiguous labels, broadening the scope of conformal inference beyond classical p-value methods. The method is validated through experiments on the FEMNIST dataset, demonstrating superior coverage and adaptability in dynamic environments.
Key Results
- In batch sequential settings, the E-value method achieved 94% empirical coverage over 50 batches, surpassing the traditional 85%, with conformal set sizes remaining manageable.
- In fixed-size, data-dependent coverage scenarios, the approach provided more adaptive and tighter conformal sets, reducing average size while maintaining coverage.
- In ambiguous ground truth scenarios with multiple expert labels, the E-value approach maintained 1−α coverage, outperforming Monte Carlo-based p-value methods, especially under non-exchangeability.
Significance
This work significantly advances conformal inference by enabling valid, flexible, and sequential coverage guarantees in complex, real-world scenarios. It addresses key limitations of classical p-value-based methods, such as non-exchangeability, data drift, and label ambiguity. The theoretical foundation via Ville’s inequality and supermartingales offers robust guarantees, fostering applications in healthcare, manufacturing, and finance where data streams are prevalent. The approach paves the way for reliable, adaptive uncertainty quantification in dynamic environments, crucial for decision-making under uncertainty.
Technical Contribution
The main technical innovation is the construction of supermartingales based on E-values (formula 4), which generalize p-value-based methods. This enables non-asymptotic, anytime-valid guarantees under non-exchangeability and distribution shifts. The integration of Ville’s inequality with supermartingales provides a rigorous, flexible framework for sequential coverage. The method also extends to ambiguous labels, offering a unified approach for diverse complex settings, and opens avenues for further theoretical and computational improvements.
Novelty
This is the first systematic incorporation of E-values into conformal prediction, enabling non-exchangeable, sequential, and ambiguous label scenarios with rigorous coverage guarantees. Unlike traditional p-value methods, the E-value approach leverages supermartingale theory, providing stronger, more adaptable guarantees, and broadening the applicability of conformal inference to real-world, dynamic data streams.
Limitations
- The supermartingale construction relies on accurate modeling of data exchangeability within batches; deviations may weaken guarantees.
- Computational complexity increases with high-dimensional data and complex models, potentially limiting scalability.
- Experimental validation is primarily on image datasets; applicability to other domains requires further testing.
Future Work
Future research will focus on optimizing supermartingale constructions for high-dimensional and non-linear models, integrating deep learning architectures, and extending the framework to multi-task and heterogeneous data environments. Additionally, exploring adaptive hyperparameter tuning and computational efficiency will be key to broader deployment.
AI Executive Summary
Conformal prediction has become a vital tool for uncertainty quantification, especially in machine learning applications. Traditional methods rely heavily on p-values, which assume data exchangeability and static environments. However, real-world scenarios often involve data streams, distribution shifts, and ambiguous labels, challenging these assumptions. To address these limitations, this paper introduces conformal e-prediction based on E-values, leveraging Ville’s inequality to construct supermartingales that guarantee coverage at any time point.
The core innovation lies in defining nonnegative E-variables (formula 4), which are averaged to form supermartingales. These supermartingales serve as the foundation for sequential coverage guarantees, even under non-exchangeability and data drift. The method's flexibility allows it to handle batch arrivals, adaptive coverage control, and ambiguous labels—scenarios where classical p-value methods struggle.
Experimental validation on the FEMNIST dataset demonstrates the effectiveness of the approach. Over 50 batches, the E-value method achieved 94% coverage, outperforming traditional methods that hovered around 85%. The conformal sets remained reasonably small, indicating practical usability. In complex label scenarios, the method maintained the desired coverage, confirming its robustness.
This work significantly broadens the applicability of conformal inference, providing a rigorous, adaptable framework suitable for modern, dynamic data environments. It opens new avenues for reliable uncertainty quantification in healthcare, manufacturing, and finance, where data streams are ubiquitous. Future directions include optimizing supermartingale constructions, extending to deep learning models, and tackling multi-task and heterogeneous data challenges, promising a transformative impact on real-time decision-making under uncertainty.
Deep Analysis
Background
Conformal prediction, introduced by Vovk et al. (2005), has evolved as a distribution-free method for constructing predictive sets with finite-sample guarantees. Its core relies on ranking nonconformity scores and controlling the rank distribution under exchangeability assumptions. Traditional approaches predominantly use p-values, which are simple but limited to static, exchangeable data. Recent advances have sought to relax these assumptions, incorporating local or conditional guarantees, but still face challenges in dynamic, non-exchangeable environments. The advent of E-values (Balinsky et al., 2024) offers a promising alternative, enabling sequential, nonparametric inference via supermartingales. This paper builds on this foundation, aiming to extend conformal prediction to more complex, real-world settings where data arrive sequentially, and labels may be ambiguous or non-exchangeable.
Core Problem
The main challenge is to develop conformal methods that remain valid in non-exchangeable, streaming, and ambiguous label settings. Existing p-value-based methods cannot guarantee coverage when data are non-i.i.d. or when the data distribution shifts over time. Moreover, they lack flexibility for adaptive, data-dependent coverage control and cannot handle scenarios with multiple, conflicting labels. These limitations hinder their deployment in applications like online monitoring, medical diagnosis, and quality control, where data are inherently dynamic and labels uncertain.
Innovation
This work introduces a novel framework that leverages E-values to construct supermartingales, enabling anytime-valid, sequential coverage guarantees. Key innovations include: 1) defining E-variables (formula 4) as averages of scores, 2) applying Ville’s inequality to these supermartingales for probabilistic control, 3) extending the approach to ambiguous labels with multiple experts, and 4) demonstrating practical effectiveness through experiments. Unlike traditional p-value methods, the E-value approach does not require exchangeability, making it suitable for non-stationary environments. It also provides stronger, adaptive guarantees, supporting complex scenarios like batch arrivals and label ambiguity.
Methodology
- �� Define E-variables (formula 4) as the average of nonconformity scores, ensuring E[E] ≤ 1 under exchangeability.
- �� Construct a filtration {Ft} that includes all observed data up to batch t.
- �� Build a supermartingale process Mt by taking the product of E-variables across batches.
- �� Apply Ville’s inequality to Mt, establishing that the probability of exceeding 1/α at any time is bounded by α.
- �� Use the supermartingale to define conformal sets (formula 5) that guarantee coverage at all times.
- �� Extend the framework to ambiguous labels by adjusting the E-variable to incorporate multiple expert predictions, maintaining the supermartingale property.
Experiments
Experiments utilize the FEMNIST dataset, simulating sequential batch arrivals from 50 writers. The model is a simple CNN inspired by LeNet, trained for 87.6% accuracy. During testing, calibration scores are computed per batch, and conformal sets are constructed using the supermartingale approach. The key hyperparameter is α=0.15, with repeated trials to assess coverage. Results show an empirical coverage of 94%, with conformal set sizes mostly manageable. Comparisons with traditional p-value methods highlight the superior sequential coverage and robustness of the E-value approach, especially under data shifts and label ambiguity.
Results
- �� The E-value based method achieved 94% coverage over 50 batches, surpassing the 85% target and traditional p-value methods.
- �� Conformal sets remained reasonably small, demonstrating practical usability.
- �� In ambiguous label scenarios, the method maintained the desired coverage, validating its robustness in non-exchangeable, complex environments.
Applications
Applicable in online healthcare diagnostics, real-time quality control, and financial risk monitoring, where data arrive sequentially and labels may be uncertain. The method supports adaptive, data-dependent coverage control, making it suitable for dynamic, high-stakes environments. It can be integrated with deep learning models for large-scale deployment, enabling reliable, real-time uncertainty quantification.
Limitations & Outlook
The supermartingale construction depends on the assumption of within-batch exchangeability; violations may weaken guarantees. Computational costs increase with high-dimensional data and complex models. The current validation is limited to image datasets; broader application requires further testing and adaptation.
Plain Language Accessible to non-experts
Imagine you’re managing a factory that produces different kinds of toys. Each day, you check a few toys to see if they meet quality standards. Traditional methods are like using a simple ruler—if the toy fits, it’s good; if not, it’s bad. But this ruler doesn’t adapt if the factory changes how it makes toys or if some toys are more complicated. Now, scientists have invented a smart measuring tool called E-value. This tool can learn from each batch of toys, adjusting itself to new conditions. It keeps track of quality over time, even if the factory’s process shifts or if some toys are tricky to judge. This way, you can always be confident that most toys meet the standards, no matter how the factory changes. It’s like having a super-smart quality inspector that never gets fooled, ensuring your toys are safe and good every day.
ELI14 Explained like you're 14
Imagine you’re playing a game where you guess if a secret number is in a certain range. Normally, you’d use a fixed rule—like a ruler—to check if your guess is right. But what if the game keeps changing, or some guesses are more uncertain? Scientists made a new kind of ruler called E-value. It can change as you play, keeping track of how good your guesses are, even if the game changes suddenly. This ruler can tell you, at any moment, whether you’re still on track or if you need to be more careful. It’s like having a magic score that updates itself, so you always know if you’re winning or losing, no matter how tricky the game gets. That way, you can keep playing confidently, knowing your chances are always fair and accurate.
Glossary
E-value (E值)
A nonnegative random variable with expected value ≤ 1, used to build supermartingales for sequential coverage guarantees. Technically, it’s defined as the average of nonconformity scores (公式4).
在本文中，E值用来替代p值，构建超鞅过程，支持非交换性和动态环境的符合预测。
Supermartingale (超鞅)
A stochastic process where the conditional expectation of the next value given past is at most the current value, used to guarantee coverage over time.
本文利用超鞅构造连续监控的符合集，确保在任何时刻都满足置信水平。
Ville’s inequality (Ville不等式)
A probabilistic bound that controls the maximum deviation of a nonnegative supermartingale, ensuring the process exceeds a threshold with probability no more than α.
用于保证超鞅在任意时刻不超过预设的置信水平，关键技术之一。
Conformal prediction (符合预测)
A distribution-free method for constructing predictive sets with finite-sample guarantees, based on nonconformity scores.
本文扩展了传统p值基础的符合预测，采用E值实现更灵活的应用场景。
Ambiguous ground truth (模糊标签)
场景中标签由多个专家提供，存在不确定性和非交换性，传统方法难以保证置信水平。
本文提出基于E值的方法，有效应对多专家、多标签环境。
Open Questions Unanswered questions from this research
- 1 如何在高维复杂模型中高效构造超鞅仍是挑战，尤其在深度学习场景下超鞅的设计和调优缺乏系统性方法。未来需要研究更智能的超鞅构建策略，以适应大规模非线性模型的需求。
Applications
Immediate Applications
医疗诊断辅助
利用符合预测为患者提供诊断区间，支持连续监控和模糊标签场景，提升诊断可靠性。
制造质量控制
在生产线上实时监控产品质量，支持批量到达和漂移检测，降低检测成本。
Long-term Vision
工业大规模自动化
结合深度学习实现大规模连续监控，推动智能制造和自动化决策。
Abstract
Conformal prediction is a powerful framework for distribution-free uncertainty quantification. The standard approach to conformal prediction relies on comparing the ranks of prediction scores: under exchangeability, the rank of a future test point cannot be too extreme relative to a calibration set. This rank-based method can be reformulated in terms of p-values. In this paper, we explore an alternative approach based on e-values, known as conformal e-prediction. E-values offer key advantages that cannot be achieved with p-values, enabling new theoretical and practical capabilities. In particular, we present three applications that leverage the unique strengths of e-values: batch anytime-valid conformal prediction, fixed-size conformal sets with data-dependent coverage, and conformal prediction under ambiguous ground truth. Overall, these examples demonstrate that e-value-based constructions provide a flexible expansion of the toolbox of conformal prediction.