Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

Over the past decade, deep neural networks have achieved remarkable success across various domains, notably in image classification with architectures like ResNet, Inception, and DenseNet. These models have surpassed human-level performance on benchmark datasets such as ImageNet, leading to widespread adoption in industry and research. However, their internal mechanisms remain largely opaque, raising concerns about trust, fairness, and accountability. Early interpretability techniques focused on local feature attribution, such as saliency maps, Grad-CAM, and Integrated Gradients, which highlight influential pixels or regions for a specific prediction. While these methods provide valuable insights, they are limited to local explanations and do not capture the high-level concepts that humans naturally understand.

Recent efforts have aimed to bridge this gap by exploring the internal representations of neural networks. Techniques like concept activation maps (CAM) and network dissection have attempted to identify neurons or layers associated with semantic concepts. Bau et al. introduced concept-based explanations by training classifiers on activation patterns, laying the groundwork for the current work. Nonetheless, these approaches often lack a systematic way to quantify the influence of high-level concepts across classes and datasets. The challenge remains: how can we define, detect, and measure the importance of human-understandable concepts within complex models?

This paper addresses this challenge by proposing a formal framework that leverages linear classifiers to extract concept vectors (CAVs) from neural activations. The approach is inspired by the observation that neural networks often encode meaningful directions in their high-dimensional activation spaces, which can be associated with semantic concepts. By quantifying the sensitivity of model outputs to these directions, the authors enable a global, interpretable, and statistically validated analysis of model behavior, advancing the field of explainable AI.

Despite progress in interpretability research, existing methods primarily focus on local explanations, which are insufficient for understanding the global decision-making process of deep models. Feature attribution techniques like saliency maps are sensitive to noise, lack robustness, and do not directly relate to human concepts. Moreover, they cannot easily compare the influence of different high-level concepts or detect biases embedded within the model. The core problem is to develop a method that can define, quantify, and statistically validate the importance of human-understandable concepts in a model’s predictions, across entire classes and datasets. Achieving this requires overcoming challenges such as: 1) defining meaningful concepts in a high-dimensional, nonlinear activation space; 2) ensuring the interpretability of the derived concept vectors; 3) providing a statistically robust measure that accounts for variability across samples and runs; and 4) enabling flexible, user-defined concepts without retraining the model. Addressing these issues is crucial for building trustworthy AI systems that can be audited for biases, fairness, and alignment with human values.

The key innovation of this work is the introduction of Concept Activation Vectors (CAVs), which serve as linear representations of high-level concepts within neural network activations. Unlike prior methods that rely on local gradients or feature importance maps, CAVs are derived by training simple linear classifiers on a small set of concept exemplars, capturing the dominant direction in activation space associated with the concept. The use of directional derivatives allows quantification of how model predictions change when inputs are perturbed along these concept directions, providing a global, quantitative measure of concept influence. The framework incorporates statistical significance testing over multiple runs, ensuring robustness against spurious correlations. This approach is model-agnostic, flexible, and capable of handling arbitrary user-defined concepts, making it highly adaptable for practical interpretability and bias detection tasks. The integration of hypothesis testing with concept vectors represents a significant methodological advancement, enabling rigorous, scalable, and interpretable analysis of deep models.

• �� Define a human-interpretable concept by selecting a set of positive example inputs (e.g., images of striped textures). • Collect activations from a specific layer of the trained neural network for these examples, forming a high-dimensional activation dataset. • Gather a set of negative examples (e.g., random images) to serve as control samples. • Train a linear classifier (e.g., logistic regression or linear SVM) to distinguish between concept and non-concept activations, deriving the classifier’s normal vector as the CAV. • Normalize the CAV to unit length to facilitate directional derivative calculations. • For each input sample, compute the directional derivative of the model’s output (e.g., class logit) along the CAV vector, using the gradient of the output with respect to the layer activations. • Aggregate these derivatives across samples within a class to compute the TCAV score, which measures the proportion of inputs for which the model is positively influenced by the concept. • Perform multiple training runs of the CAV with different random subsets, applying t-tests to assess the statistical significance of the concept’s influence. • Use the significance results to filter out spurious concepts, ensuring only meaningful concepts are reported. • Visualize the concepts via sorting images based on cosine similarity to the CAV or through activation maximization techniques like Deep Dream. • Apply the framework to various layers and concepts, analyze the hierarchical learning process, and compare across models and datasets.

The experimental setup involves multiple datasets, including ImageNet, CIFAR-10, and a medical dataset for diabetic retinopathy. For each dataset, the authors select a diverse set of concepts—such as colors, textures, objects, gender, and race—by collecting representative exemplars. These exemplars are used to train CAVs at different network layers, typically convolutional or fully connected layers. The evaluation metrics include the TCAV score (fraction of inputs positively influenced by the concept), statistical significance (p-values from t-tests), and correlation with human annotations. The experiments examine the impact of sample size (30, 50, 100 images per concept) on CAV robustness, demonstrating that even small samples can produce meaningful results. Control experiments with noisy labels or random exemplars assess the specificity of the method. The authors compare TCAV with traditional feature attribution methods, showing superior global interpretability and bias detection capabilities. In the medical case, the method identifies key features influencing diagnosis, validated by domain experts, illustrating practical utility. The experiments also analyze the hierarchical learning of concepts across layers, revealing how simple features are captured in early layers and abstract concepts in deeper layers.

TCAV effectively quantifies the influence of human-defined high-level concepts on neural network predictions, with scores exceeding 0.75 for relevant concepts and statistically significant p-values (<0.01). In ImageNet, concepts like ‘stripes’ significantly impacted ‘zebra’ classification, while ‘color’ influenced ‘dog’ categories. In bias detection, TCAV uncovered that models trained on ImageNet exhibited sensitivity to ‘gender’ and ‘race’ attributes, correlating with known dataset biases. In medical applications, TCAV identified ‘blood vessels’ and ‘exudates’ as critical features for diabetic retinopathy, aligning with ophthalmological expertise. The robustness of the method was validated through experiments with limited samples (30 images), which still yielded high CAV accuracy (>85%). The statistical significance testing effectively filtered out false positives, ensuring reliable interpretations. Overall, the results demonstrate that TCAV provides a scalable, quantitative, and concept-level understanding of model behavior, surpassing traditional local attribution methods in interpretability and bias detection.

TCAV can be immediately applied in domains requiring model transparency, such as medical diagnosis, where understanding the influence of high-level features like ‘vessels’ or ‘lesions’ is critical. It supports bias detection in fairness auditing by quantifying the impact of sensitive attributes like gender or race across entire datasets. In autonomous systems, TCAV helps verify that models rely on appropriate concepts rather than spurious correlations, enhancing safety and trust. The method’s flexibility allows practitioners to define custom concepts relevant to their domain, facilitating targeted model debugging and validation. Long-term, TCAV could be integrated into automated model auditing pipelines, enabling continuous bias monitoring and concept-based explanations, ultimately fostering responsible AI deployment across industries.

The effectiveness of TCAV depends heavily on the quality and representativeness of the concept exemplars; poorly chosen samples can lead to misleading interpretations. The linear assumption underlying CAVs may not hold in highly nonlinear or complex models, limiting the fidelity of concept representations. The computational cost increases with multiple runs for statistical validation, especially in large-scale models or datasets. Additionally, defining meaningful concepts requires domain expertise, and abstract or fuzzy concepts pose challenges for exemplar collection. Future work should focus on automating concept discovery, extending to non-linear representations, and improving computational efficiency to make the approach more scalable and user-friendly.