Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings

In high-stakes domains such as fraud detection, credit assessment, and healthcare, machine learning model predictions often require human decision-makers' review. In these settings, model outputs rarely constitute final decisions. Instead, predictions are reviewed by human decision-makers operating under time, attention, and regulatory constraints. As a result, explanations are viewed as indispensable for accountability and oversight and have become a core requirement in operational ML deployments. Despite widespread adoption, the practical value of explanations in human-in-the-loop workflows remains poorly understood, often assumed rather than empirically established. Among explanation methods, local approaches grounded in cooperative game theory, most notably Shapley values, have emerged as a cornerstone by providing an axiomatic decomposition of model predictions into feature-level contributions. However, the framework has fragmented into competing formulations based on divergent assumptions about the semantics of feature absence, realized in popular implementations such as KernelSHAP, TreeSHAP, and related tools. This raises a critical evaluation question for practitioners: does the choice of formulation matter to the end-user, and do standard evaluation procedures anticipate the impact?

Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape with little consensus on practical deployment. While theoretical differences are well-documented, evaluation remains reliant on quantitative proxies whose alignment with human utility is unverified. Modern XAI evaluation relies on theoretical analysis and quantitative proxies, as well as mathematical distinctions between 'faithfulness' to the model or 'truthfulness' to the data. Yet, systematic evidence regarding how explanation methods perform against human-centered benchmarks remains scarce. Existing evaluations often focus on isolated properties and rarely stress-test these metrics against human behavior under realistic operational constraints. Furthermore, comparisons are often confounded by implementation choices, which mask the true semantic differences of the definitions themselves.

The core innovations of this paper include: 1) Proposing a unified amortized framework that eliminates implementation confounders for fair comparison of different Shapley variants; 2) Providing large-scale empirical analysis revealing the fundamental misalignment between quantitative metrics and human perception; 3) Offering evidence-based guidance for selecting Shapley values and evaluation metrics in high-risk decision systems. These innovations not only enhance the transparency and interpretability of AI systems but also mitigate the risk of automation bias.

• �� Employ a unified amortized framework to eliminate implementation confounders, enabling fair comparison of eight Shapley value variants.
• �� Conduct research across four risk datasets and a realistic fraud detection environment involving 37 professional analysts and 3,735 case reviews.
• �� This approach reveals a fundamental misalignment between quantitative metrics and human perception of clarity and decision utility.
• �� Evaluate Shapley attributions along two complementary axes: quantitative evaluation and human-in-the-loop study.
• �� Use a compact set of metrics to capture functional properties, cross-formulation agreement, and downstream analyst behavior.

The experimental design includes four risk datasets and a realistic fraud detection environment involving 37 professional analysts and 3,735 case reviews. Baselines include popular implementations like KernelSHAP and TreeSHAP. Evaluation metrics include sparsity, faithfulness, contrastivity, Deletion AUC, and Recall@3. The experiments also include ablation studies to analyze performance differences across different Shapley variants.

Experimental results show that standard quantitative metrics such as sparsity and faithfulness are decoupled from human-perceived clarity and decision utility. While explanations did not improve objective analyst performance, they consistently increased decision confidence, indicating a critical risk of automation bias in high-stakes settings. By comparing eight Shapley variants, the study finds no single variant dominates across all metrics. Fixed baseline variants perform well on Deletion AUC and Recall@3 but poorly on sparsity and contrastivity. Empirical variants show balanced performance, while Conditional Shapley deviates from this pattern, producing dense, sensitive attributions that reflect correlations rather than model behavior.

The application scenarios of this research include high-risk decision systems in financial and fraud detection domains. By revealing the misalignment between current evaluation metrics and human utility, the study provides evidence-based guidance for selecting appropriate Shapley variants and evaluation metrics. This not only enhances the transparency and interpretability of AI systems but also mitigates the risk of automation bias.

The study primarily focuses on financial and fraud detection domains, and results may not generalize to vision or language domains where feature semantics may have different dynamics. Additionally, experiments were conducted in controlled settings, unable to capture long-term effects such as learning, adaptation, or changes in institutional decision norms. Future research could extend to other domains such as vision and natural language processing to validate the utility of Shapley value variants in different applications. Additionally, developing new evaluation metrics to better predict human perception and decision utility is an important direction.