Carefree multiple testing with e-processes
Proposes a carefree multiple testing framework using e-process supremum and adjusters to control FDR under arbitrary dependence.
Key Findings
Methodology
This paper introduces the concept of taking the supremum of e-processes within multiple testing, combined with the use of adjusters to transform the maximum into a valid e-process. The approach ensures FDR control under arbitrary dependence structures. The authors analyze the limitations of e-BH in dependent settings, demonstrate how the supremum-based method overcomes these, and provide theoretical guarantees. Simulations validate that the adjusted supremum method maintains FDR below the nominal level, outperforming traditional approaches under dependence.
Key Results
- Simulations show that e-BH applied to the maximum of dependent e-processes fails to control FDR, reaching approximately 1.08×α, whereas the adjusted supremum method keeps FDR below α, confirming theoretical guarantees.
- Using specific adjusters A1 and A2, the method maintains FDR control in highly dependent scenarios with minimal power loss, demonstrating robustness.
- Results indicate that naive maximum-based e-BH is invalid under dependence, but the adjusted approach effectively restores FDR control while preserving detection power, validated through extensive simulations.
Significance
This work advances the theory of multiple testing with e-processes, providing a robust framework for FDR control in complex dependence environments. It addresses a critical gap where classical methods fail, enabling reliable inference in high-dimensional, dependent data such as genomics and neuroscience. The approach offers a practical solution for ongoing data collection scenarios, broadening the applicability of e-process-based inference and fostering new research directions.
Technical Contribution
The paper's main technical innovation is the integration of the supremum of e-processes with adjusters to produce a valid e-process that guarantees FDR control under arbitrary dependence. It extends the e-BH procedure by incorporating these elements, providing rigorous proofs and simulation validation. This approach generalizes existing methods, allowing for flexible dependence structures and continuous monitoring, and introduces new theoretical guarantees for FDR super-control.
Novelty
This is the first comprehensive framework to leverage the supremum of e-processes combined with adjusters for multiple testing under arbitrary dependence. Unlike prior work limited to independent or positively dependent e-values, this method ensures FDR control universally. The innovative use of the supremum and adjusters distinguishes it from traditional p-value or e-value based methods, representing a significant conceptual leap.
Limitations
- Adjusters introduce conservativeness, leading to reduced power, especially in weak signal scenarios.
- Computational complexity increases with the number of hypotheses and dependence structure, limiting scalability.
- Performance in ultra-high-dimensional settings or with extreme dependence remains to be fully explored.
Future Work
Future research will focus on designing less conservative adjusters to improve power, extending the framework to nonparametric and high-dimensional dependence models, and integrating with machine learning techniques for adaptive inference. Further, exploring computational efficiency and real-data applications in genomics and neuroimaging will be key directions.
AI Executive Summary
In the era of big data, multiple hypothesis testing faces the challenge of controlling false discoveries amidst complex dependence structures. Traditional methods like Benjamini-Hochberg (1990) work well under independence but falter when dependencies are strong or unknown. The advent of e-processes, which allow continuous monitoring and data collection, has opened new avenues for flexible inference. However, existing e-value based methods such as e-BH struggle with dependence, risking inflated false discovery rates.
This paper introduces a novel approach that leverages the supremum of e-processes, combined with carefully designed adjusters, to achieve robust FDR control under arbitrary dependence. By focusing on the maximum of the e-process over time, the authors develop a framework that guarantees the FDR-sup (a conservative version of FDR) remains below the desired level. The key innovation lies in transforming the maximum into a valid e-process via adjusters, ensuring theoretical guarantees without restrictive dependence assumptions.
Simulations demonstrate that naive maximum-based methods fail to control FDR, often exceeding the nominal level, but the adjusted supremum approach maintains control effectively. Although the conservativeness of adjusters can reduce power, the method provides a reliable foundation for sequential, dependent data analysis. This work significantly broadens the applicability of e-processes, enabling their use in complex, real-world scenarios such as genomics, neuroimaging, and online experiments. Future research aims to refine adjusters for better power, extend to high-dimensional settings, and implement scalable algorithms, promising a new era of dependable, continuous inference in dependent data environments.
Deep Dive
Abstract
E-processes enable hypothesis testing with ongoing data collection while maintaining Type I error control. However, when testing multiple hypotheses simultaneously, current $e$-value based multiple testing methods such as e-BH are not invariant to the order in which data are gathered for the different $e$-processes. This can lead to undesirable situations, e.g., where a hypothesis rejected at time $t$ is no longer rejected at time $t+1$ after choosing to gather more data for one or more $e$-processes unrelated to that hypothesis. We argue that multiple testing methods should always work with suprema of $e$-processes. We provide an example to illustrate that e-BH does not control the FDR, at level $α$ when applied to suprema of $e$-processes. From the same example we see that the FWER is not controlled with averaging, and also closed e-BH does not control the FDR. We show that adjusters can be used to ensure FDR-sup control with e-BH under arbitrary dependence.