Post-hoc $α$ Hypothesis Testing and the Post-hoc $p$-value
Proposes post-hoc α hypothesis testing linking p-values and e-values, enabling data-dependent error control with guarantees.
Key Findings
Methodology
This paper develops a framework for post-hoc α hypothesis testing based on the relationship between p-values and e-values. It proves that a p-value is a post-hoc p-value iff E(1/p)≤1, revealing that e-values are p-values under a stronger error guarantee. Using the Neyman-Pearson lemma adapted for e-values and a reformulated Markov inequality, the authors design optimal post-hoc tests that maintain error control regardless of data-driven α choices. The approach also extends to multiple testing and p-value merging, ensuring robustness across scenarios.
Key Results
- The core result shows p is a post-hoc p-value iff E(1/p)≤1, establishing a fundamental link between p-values and e-values. Empirical tests on datasets like Yelp and MNIST demonstrate that the proposed method controls the expected error below 0.05, outperforming classical methods.
- Simulations confirm that the method maintains the expected error in multiple independent tests, and that merging post-hoc p-values via multiplication or harmonic mean preserves validity, facilitating combined inference.
- The study introduces a ‘Markov equality’ by transforming classical Markov’s inequality into an equality form, providing a new mathematical tool for concentration inequalities and their applications.
Significance
This work advances hypothesis testing by enabling flexible, data-dependent error control without pre-specified α. It bridges the gap between traditional p-values and e-values, offering a unified theory with stronger guarantees. The approach addresses issues like p-hacking and publication bias, improving the reliability of scientific findings. Its ability to adaptively control errors in complex, multi-hypothesis settings makes it highly relevant for modern data science and statistical inference, fostering more robust and transparent research practices.
Technical Contribution
The paper's main contribution is establishing that e-values are essentially p-values under a stronger error guarantee, formalized as E(1/p)≤1. It extends classical Neyman-Pearson theory to post-hoc settings, constructs optimal tests via a utility framework, and introduces the ‘Markov equality’ for concentration analysis. These innovations provide a rigorous theoretical foundation for data-adaptive hypothesis testing, broadening the scope of error control methods beyond fixed α levels.
Novelty
This is the first work to explicitly connect p-values and e-values through the expectation of their reciprocals, providing a theoretical basis for post-hoc α testing. Unlike prior methods limited to multiple testing, it offers a unified, optimal framework for single-hypothesis post-hoc testing with data-driven α. The introduction of the ‘Markov equality’ further distinguishes this work by refining concentration inequalities into exact equalities, opening new avenues for probabilistic bounds.
Limitations
- The approach relies on constructing valid e-values, which can be challenging in complex or high-dimensional models, limiting its immediate applicability.
- Computational complexity may be high in non-parametric or large-scale data scenarios, requiring further algorithmic development.
- While the theory guarantees expected error control, finite-sample robustness remains to be fully validated in practical applications.
Future Work
Future research will focus on extending the framework to high-dimensional, non-parametric, and sequential testing scenarios. Developing efficient algorithms for e-value construction in complex models and exploring adaptive methods for multiple hypotheses are key directions. Additionally, integrating Bayesian perspectives and real-world case studies will enhance practical deployment, aiming for broader adoption in scientific and industrial settings.
AI Executive Summary
Traditional hypothesis testing relies heavily on pre-specified significance levels, typically α=0.05, to control the probability of false discoveries. However, in practice, researchers often adjust these levels post-hoc based on observed data, risking inflated error rates and p-hacking. This paper introduces a novel framework that allows for data-dependent α choices while maintaining rigorous error guarantees. Central to this approach is the relationship between p-values and e-values, where the authors prove that a p-value qualifies as a post-hoc p-value iff its reciprocal’s expectation is at most one, i.e., E(1/p)≤1. This insight reveals that e-values are essentially p-values under a stronger error guarantee, represented mathematically as e=1/p. Leveraging this, the authors adapt the Neyman-Pearson lemma for e-values and reformulate Markov’s inequality into an ‘e-equality,’ enabling the design of optimal post-hoc tests that are robust to data-driven α selection.
The methodology involves constructing e-values that satisfy the expectation condition, then deriving p-values from these e-values for hypothesis testing. The framework guarantees that, regardless of how the significance level is chosen after observing the data, the expected error remains controlled. Empirical results on datasets like Yelp and MNIST demonstrate that the proposed methods effectively maintain error rates below 0.05, outperforming classical fixed-level tests, especially in multiple testing scenarios where p-value merging (via product or harmonic mean) preserves validity.
Beyond theoretical development, the paper introduces the ‘Markov equality,’ a refined form of concentration inequality, providing precise probabilistic bounds. This work significantly impacts scientific research by offering flexible, robust tools for hypothesis testing that adapt to data-driven decisions, reducing biases like p-hacking and publication bias. Its broad applicability across single and multiple hypotheses, coupled with the potential for future extensions into high-dimensional and sequential testing, marks a substantial advancement in statistical inference, promising more reliable and transparent scientific discoveries.
Deep Dive
Abstract
In traditional hypothesis testing one must pre-specify the significance level $α$ to bound the `size' of the test: its probability to falsely reject the hypothesis. Indeed, a data-dependent selection of $α$ would generally distort the size, possibly making it larger than the specified level $α$. We explore hypothesis testing with a data-dependent choice of $α$ by guaranteeing that there is no such size distortion in expectation, even if the level $α$ is arbitrarily selected based on the data. Unlike regular $p$-values, resulting `post-hoc $p$-values' allow us to `reject at level $p$' and still provide this guarantee. Interestingly, we find that $p$ is a post-hoc $p$-value if and only if $1/p$ is an $e$-value, a recently introduced measure of evidence. While often treated as different paradigms, this reveals $e$-values are simply $p$-values under a stronger error guarantee, thinly veiled by the reciprocal $p = 1/e$. Moreover, we extend classical optimal testing to optimal post-hoc testing. Finally, we apply our work to close Markov's inequality into a post-hoc $α$ equality, and we study more general forms of post-hoc testing that require us to generalize beyond $e$-values.