Instrument-Armed Bandits

The multi-armed bandit (MAB) problem has been a central topic in sequential decision-making, with classical algorithms like UCB and Thompson Sampling achieving near-optimal regret bounds in stationary reward environments. Recent advances incorporate contextual information and Bayesian methods, but most assume full control over treatment assignment. In real-world applications involving humans, non-compliance—where subjects do not follow assigned treatments—is widespread, introducing bias and complicating causal inference. Econometrics introduced instrumental variables (IV) to address such issues, with methods like 2SLS providing consistent causal effect estimates. However, integrating IV techniques into reinforcement learning remains underdeveloped. This paper bridges this gap by formalizing the instrument-armed bandit (IAB) model, explicitly modeling non-compliance, and analyzing its impact on regret and learning efficiency. It extends the theoretical understanding of causal bandits, emphasizing the importance of aligning learning objectives with causal estimands, especially in sensitive domains like healthcare.

In practical scenarios, treatment assignment (arm pull) does not guarantee actual treatment application due to non-compliance, which varies across individuals and contexts. This discrepancy leads to challenges in estimating true causal effects, as standard algorithms optimize for observed rewards rather than causal impact. Existing methods fail to account for the bias introduced by non-compliance, resulting in suboptimal decision policies. The core problem is how to define appropriate regret metrics that reflect causal objectives, and how to design algorithms capable of learning treatments with maximal causal effects despite incomplete and biased feedback. High non-compliance rates exacerbate these issues, making it difficult to reliably identify optimal interventions, especially when the treatment effects are heterogeneous.

The paper's key innovations include: 1) formalizing the instrument-armed bandit (IAB) model that captures non-compliance effects; 2) introducing multiple regret measures tailored to causal inference, such as compliers’ regret and latent class regret; 3) integrating causal inference techniques, notably 2SLS, into bandit algorithms to estimate and optimize causal effects; 4) deriving theoretical regret bounds under assumptions like treatment effect homogeneity; 5) developing practical algorithms (e.g., 2SLS-ϵ-decay) that adapt exploration based on causal estimates. These contributions enable the design of algorithms that focus on causal rather than purely reward-based objectives, addressing a critical gap in reinforcement learning for human-centered applications.

• �� Model the non-compliance scenario by defining the treatment vector and compliance types, incorporating observed rewards and unobserved compliance. • Define multiple regret metrics aligned with causal goals, including intent-to-treat, static, latent class, and compliers’ regret. • Use instrumental variable techniques (like 2SLS) to estimate the causal effect of treatments, integrating these estimates into bandit algorithms. • Design algorithms (e.g., 2SLS-ϵ-衰减) that balance exploration and exploitation based on causal estimates, with theoretical analysis deriving regret bounds. • Prove that under homogeneity assumptions, the algorithms achieve logarithmic regret, and validate through simulations. • Conduct experiments on synthetic and real clinical data, comparing with baseline algorithms, analyzing the impact of non-compliance on regret and effect estimation.

Simulations involve multiple arms with varying non-compliance rates (10%, 30%, 50%), assessing regret and causal effect estimation accuracy. Baselines include standard UCB and Thompson Sampling. Metrics include cumulative regret, bias in effect estimates, and confidence interval coverage. Hyperparameters are tuned for exploration-exploitation balance. Ablation studies examine the impact of different regret definitions and assumptions like homogeneity. Real data from clinical trials validate the model's ability to identify treatments with maximum causal effects, demonstrating robustness under realistic non-compliance scenarios.

Proposed algorithms significantly outperform traditional UCB, reducing regret by over 50% in high non-compliance settings. They accurately estimate the maximum causal effect, with bias reduced by 40-60%. Simulation results confirm the importance of selecting the appropriate regret metric—algorithms targeting causal effects outperform reward-based methods in effect estimation. Real data analysis shows improved treatment effect identification and bias correction, validating the theoretical guarantees. The results demonstrate the practical viability of integrating causal inference into reinforcement learning frameworks.

Applicable in clinical trials, policy evaluation, personalized medicine, and adaptive recommendation systems where non-compliance is prevalent. Enables more accurate estimation of treatment effects, leading to fairer and more effective interventions. Prerequisites include observable rewards, treatment assignment data, and assumptions like homogeneity or linearity. The framework can be extended to large-scale, high-dimensional causal structures, supporting real-time decision-making in healthcare and social policy.

Assumes reward distributions are stationary and satisfy specific regularity conditions, limiting applicability in dynamic environments. Computational complexity increases with the number of arms and causal structure complexity, requiring further optimization. High non-compliance rates pose challenges for regret control, especially when effects are heterogeneous. Future work should address non-stationarity, scalability, and robustness to model misspecification.