Bringing closure to FDR control: beating the e-Benjamini-Hochberg procedure

The evolution of multiple hypothesis testing has seen classical procedures like BH become standard for FDR control under independence. However, real-world data often exhibit complex dependence, limiting these methods' effectiveness. The introduction of e-values provided robustness against dependence, leading to methods like eBH, which guarantees FDR control under arbitrary dependence. Nonetheless, existing techniques lack a unified theoretical framework and flexibility for post-hoc error metric selection. Recent studies have explored adaptive and randomized approaches, but they often involve trade-offs between power and control. This paper addresses these gaps by establishing a general closure principle grounded in e-values, aiming to unify and improve FDR control across diverse settings.

Current FDR control methods struggle with complex dependence, multi-source data, and the need for flexible error metrics. Traditional procedures either lack robustness or are conservative, reducing power. The challenge is to develop a unified, scalable framework that guarantees strict error control, adapts to various dependence structures, and allows post-hoc decision-making. Achieving this requires reconciling theoretical rigor with computational efficiency, especially for high-dimensional data, which remains an open problem in the field.

The paper introduces a universal closure principle that applies to all expectation-based error metrics, expressed via e-values for intersection hypotheses. This unifies multiple testing procedures under a single theoretical umbrella. The development of the closed eBH and BY procedures, supported by a dynamic programming algorithm, enables scalable, high-power testing under arbitrary dependence. The framework also allows post-hoc selection of significance levels and error metrics, providing unprecedented flexibility. These innovations fundamentally extend the theoretical landscape of multiple testing, offering a robust, adaptable, and computationally feasible approach.

• �� Define e-values for each hypothesis and intersection hypotheses, capturing dependence structures. • Formulate candidate discovery sets based on supremum expectations of e-values, ensuring FDR control. • Develop the closed eBH procedure by maximizing the size of the rejection set within the candidate set, using a dynamic programming algorithm for efficiency. • Extend to the closed BY procedure, dominating the original BY method under arbitrary dependence. • Prove that all FDR controlling procedures can be represented as e-closed procedures, enabling post-hoc error metric selection. • Incorporate post-hoc FDP control via supremum-based e-values, enhancing flexibility. • Validate theoretical guarantees through rigorous proofs and extensive simulations, demonstrating improved power and control.

Simulations involved generating dependent e-values under various correlation structures, including block dependence and arbitrary correlation matrices. Baselines included traditional BH, BY, and eBH methods. Metrics assessed were FDR, power (number of true discoveries), and computational time. Hyperparameters such as significance level α and dependence strength were varied. Additional experiments tested robustness to e-value estimation errors and dependence misspecification. Results consistently showed the proposed methods achieved lower FDR and higher power, especially in complex dependence scenarios, confirming theoretical advantages. Scalability was validated on datasets with up to 10,000 hypotheses, demonstrating practical feasibility.

The closed eBH method achieved over 20% higher rejection counts than original eBH in simulated dependent data, maintaining FDR below 0.05. Compared to BY, it showed lower FDR and increased true discoveries across all dependence structures. The dynamic programming algorithm computed rejection sets efficiently, with runtime scaling as O(K^3). The post-hoc error metric selection enabled adaptive control, matching or exceeding fixed-level procedures. These results confirm the framework’s ability to improve power while ensuring rigorous error control, addressing key limitations of existing methods.

Applicable in high-dimensional genomics, neuroimaging, and multi-source data integration, where dependence is complex and control guarantees are critical. The framework supports adaptive, data-driven decision-making, enabling researchers to tailor error metrics post-hoc. It also benefits industries requiring reliable multiple testing procedures, such as drug discovery, finance, and machine learning, by providing scalable, robust tools that adapt to various dependence structures and data complexities.

The computational complexity of O(K^3) may limit scalability for extremely large datasets, necessitating further optimization. Dependence modeling assumptions, while relaxed, may still impact control guarantees under highly intricate dependence. Accurate e-value estimation remains challenging in practice, especially with limited data or model misspecification. Future work should focus on algorithmic acceleration, dependence robustness, and broader applicability to continuous or multi-stage testing scenarios.