Conformal bandits: bringing statistical validity and reward efficiency under weak arm separability

The evolution of multi-armed bandit (MAB) algorithms has been driven by the need for efficient exploration-exploitation strategies in uncertain environments. Classical methods like UCB and Thompson Sampling rely on Hoeffding and Bayesian bounds, respectively, providing strong guarantees under large-arm gaps. However, these approaches falter in weak-arm scenarios with minimal reward differences, especially when reward distributions are heteroscedastic or non-stationary. Recent advances in conformal prediction (CP) have established non-parametric, finite-sample coverage guarantees for predictive intervals, primarily in regression and classification tasks. Yet, their integration into sequential decision-making, particularly in MAB settings, remains underexplored. Addressing this gap, the paper leverages CP's flexibility to construct statistically valid reward bounds, enhancing robustness and adaptivity in complex environments like financial markets, where regime shifts and subtle differences challenge traditional algorithms.

The core challenge lies in the limitations of existing UCB strategies under weak arm separability and low signal-to-noise ratios. Classical confidence bounds become overly conservative or unstable, leading to slow learning or premature exploitation. In real-world applications such as portfolio management, medical trials, and online advertising, reward differences are often negligible, and reward distributions exhibit heteroscedasticity and non-stationarity. These issues result in increased regret and unreliable uncertainty quantification. The fundamental problem is to develop a sequential decision framework that guarantees finite-sample statistical validity, adapts to environment changes, and maintains reward efficiency, especially in high-noise, low-gap regimes.

The key innovations include: 1) embedding conformal prediction within the bandit framework to produce finite-sample valid prediction intervals, replacing Hoeffding bounds; 2) developing Conformal UCB, which combines conformal bounds with a risk-sensitive parameter λ for flexible exploration strategies; 3) utilizing split conformal prediction and conformal quantile regression (CQR) to handle heteroscedastic and skewed reward distributions; 4) integrating hidden Markov models (HMM) to detect market regimes and dynamically adjust exploration. These components collectively address the limitations of classical methods, providing statistically valid, adaptive, and risk-aware decision policies that perform well in complex, non-stationary environments.

• �� Construct future reward prediction intervals using split conformal prediction, ensuring finite-sample coverage under exchangeability.
• �� Apply conformal quantile regression (CQR) to model heteroscedastic reward distributions, capturing asymmetries and non-stationarity.
• �� Replace Hoeffding-based confidence bounds in UCB with conformal intervals, forming Conformal UCB.
• �� Introduce a risk parameter λ ∈ [0,1], convexly combining upper and lower conformal bounds to interpolate between optimistic and conservative policies.
• �� Incorporate hidden Markov models (HMM) to identify market regimes, dynamically adjusting exploration based on inferred states.
• �� Evaluate the approach through simulations on synthetic data with controlled gaps and noise, and real financial data, measuring regret, coverage, Sharpe ratio, and drawdowns.
• �� Conduct ablation studies to isolate the impact of CP, HMM, and λ tuning, ensuring robustness and adaptability.

• �� Synthetic experiments simulate multi-arm environments with varying reward gaps, noise levels, and non-stationarity, comparing Conformal Bandits with classical UCB and risk-sensitive variants.
• �� Financial data experiments utilize stock market datasets, applying HMM to detect regimes, and evaluating portfolio strategies based on the proposed method.
• �� Metrics include cumulative regret, finite-sample coverage probability, Sharpe ratio, maximum drawdown, and exploration-exploitation balance.
• �� Hyperparameters such as the conformal calibration set size, λ, and HMM parameters are tuned via cross-validation.
• �� Results demonstrate consistent coverage guarantees, reduced regret (over 20%), and improved financial metrics, validating the approach’s effectiveness.

• �� In controlled simulations, Conformal Bandits achieved over 95% coverage with 20% lower average regret than Hoeffding-based UCB in weak-arm scenarios.
• �� Financial experiments showed a 15% increase in Sharpe ratio and 10% reduction in maximum drawdown when combining HMM with conformal bounds, outperforming baseline strategies.
• �� Ablation studies confirmed that CP and HMM contributions are critical for robustness, especially under non-stationarity and small reward gaps, with a 10% boost in risk-adjusted returns.
• �� The method maintains stable exploration-exploitation trade-offs across diverse environments, demonstrating broad applicability.

• �� Immediate application in quantitative finance for dynamic portfolio optimization, providing statistically valid risk-adjusted strategies.
• �� Healthcare, for adaptive clinical trial designs where treatment effects are subtle and environment shifts occur.
• �� Online recommendation systems, enhancing robustness against distribution shifts and subtle user preference differences.

• �� Computational overhead of conformal prediction, especially with high-dimensional data or real-time constraints, requires further optimization.
• �� Assumption of data exchangeability may limit applicability in highly dependent or non-stationary environments.
• �� Performance in extreme market downturns or highly volatile settings needs further validation, and model extensions are necessary for broader robustness.