Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
Proposes VH-CBM, combining VLM embeddings and minimal annotations via Gaussian Processes to enhance concept accuracy and calibration.
Key Findings
Methodology
This paper introduces the VH-CBM framework, leveraging pre-trained vision-language models (VLMs) embeddings and Bayesian Gaussian Processes (GPs) to propagate limited expert annotations. The process involves: • extracting input embeddings from VLMs; • training per-concept GPs in embedding space using sparse annotations; • employing variational inference to obtain mean and variance of concept activations, improving calibration; • feeding these statistics into a linear classifier for task prediction. The approach balances interpretability with broad applicability, supporting active learning to reduce annotation costs.
Key Results
- On four datasets, VH-CBM with only 1% labeled data significantly outperforms pure VLM-guided CBMs, with at least 15% higher F1 scores and 30% lower calibration error.
- In Shape3d and CelebA, concept prediction accuracy reached 85% and 88%, respectively, surpassing VLM methods’ 70% and 75%, with task F1 scores improving by 5-8%.
- Increasing annotations further improves concept accuracy, and active learning reduces labeling effort while maintaining high performance.
Significance
This work addresses the critical trade-off between interpretability and applicability in concept bottleneck models. By integrating VLMs with minimal expert supervision through Gaussian Processes, it enhances concept quality and calibration, enabling transparent AI in high-stakes fields like healthcare and autonomous driving. The method reduces reliance on costly annotations while maintaining high task performance, advancing practical deployment of explainable AI.
Technical Contribution
The key innovation lies in embedding space-based Gaussian Process modeling, which propagates sparse expert labels with uncertainty estimates, improving concept accuracy and calibration. This Bayesian approach supports active learning and scales efficiently with the number of concepts via inducing points, offering a new paradigm for combining large-scale multimodal models with probabilistic calibration.
Novelty
This is the first application of Gaussian Processes within VLM-guided concept bottleneck models, enabling label propagation in embedding space. It effectively bridges the gap between weak supervision and high-quality concept learning, outperforming existing methods that rely solely on VLM responses or full supervision, thus setting a new standard for interpretability with minimal annotation.
Limitations
- The approach depends on the stability of VLM embedding spaces; in out-of-distribution or highly biased domains, performance may decline.
- Gaussian Process inference, although scalable with inducing points, still faces computational challenges with very large datasets.
- While few annotations suffice, complex tasks may require more expert input, and annotation cost remains a consideration.
Future Work
Future directions include integrating multi-modal data, enhancing robustness in domain-shift scenarios, and developing active learning strategies to further minimize annotation effort. Extending the framework to multi-task settings and real-time applications will also be explored, aiming to make explainable AI more practical and scalable.
AI Executive Summary
In the rapidly evolving field of artificial intelligence, achieving both high performance and interpretability remains a fundamental challenge. Traditional concept bottleneck models (CBMs) offer transparency by relying on high-level human-understandable concepts, but their dependence on costly expert annotations limits widespread adoption. Recent advances leverage vision-language models (VLMs) to generate weak concept supervision, significantly broadening applicability. However, these VLM-guided CBMs often produce less accurate and poorly calibrated concepts, undermining their interpretability. Addressing this, the present study introduces VH-CBM, a hybrid framework that combines the global embedding space of VLMs with a small set of expert annotations, propagated through Bayesian Gaussian Processes. This approach effectively enhances concept accuracy and calibration, even with minimal supervision. Extensive experiments across four datasets—including Shape3d, CelebA, CUB, and Derma—demonstrate that VH-CBM outperforms existing VLM-guided models, achieving at least 15% higher F1 scores on concept prediction with only 1% labeled data. The Bayesian nature of GPs provides uncertainty estimates, supporting active learning strategies that further reduce annotation costs. Importantly, the method maintains high task accuracy, making it suitable for high-stakes applications like medical diagnosis and autonomous driving. The framework’s scalability and efficiency are validated through the use of inducing points and parallel training. Overall, VH-CBM advances the field by bridging the gap between interpretability and applicability, offering a practical solution for deploying transparent AI systems with minimal annotation effort. Future work will explore multi-modal data fusion, domain robustness, and real-time deployment, pushing the boundaries of explainable AI in complex real-world scenarios.
Deep Analysis
Background
近年来，深度学习模型在性能上取得巨大突破，但黑箱特性限制了其在高风险领域的应用。概念瓶颈模型（CBM）通过引入高层次概念实现可解释性，代表性工作包括Koh等（2020）提出的两阶段训练架构。随着VLM的出现，利用大规模预训练模型进行弱监督标注，极大拓展了CBM的适用范围，但其概念质量受VLM响应不稳定影响，导致解释性下降。如何在降低标注成本的同时，保证概念的准确性和校准，成为研究热点。
Core Problem
核心问题在于，纯CBM依赖昂贵的高质量标注，限制了其广泛应用；而纯VLM引导的模型虽然应用广泛，但概念的准确性和校准水平不足，影响模型的可解释性。两者之间存在明显的折中关系，如何在少量专家标注下提升概念的准确性和校准，成为亟待解决的难题。
Innovation
本研究提出VH-CBM，主要创新点包括：• 利用预训练VLM的全局嵌入空间作为信息基础；• 在嵌入空间中引入贝叶斯高斯过程（GP），通过专家标注传播信息，提升概念的准确性和校准；• 支持少量专家标注与GP传播结合，兼顾模型可解释性与广泛适用性。这一方案突破了纯VLM或传统CBM的局限，兼具解释性和实用性。
Methodology
- �� 提取输入的VLM嵌入z作为基础特征；• 在嵌入空间中为每个概念训练多类或二元高斯过程（GP）分类器，利用少量专家标注进行传播；• 采用变分推断（ELBO最大化）训练GP，获得每个概念的均值与方差，提升校准；• 将GP输出的统计信息输入线性层，进行最终任务预测；• 支持主动学习，通过GP不确定性指标选择最具信息量的样本进行标注，优化标注效率。
Experiments
在Shape3d、CelebA、CUB和Derma等数据集上，比较VH-CBM与纯VLM引导模型和全标注CBM。采用F1、ROC-AUC、校准误差等指标，验证少量标注下的性能提升。设计包括不同标注比例、主动学习策略、多VLM背骨（CLIP、DINO）等，确保结果的稳健性和适用性。实验还分析了标注量变化对概念准确性和任务性能的影响。
Results
在只标注1%的数据时，VH-CBM的概念F1提升至少15%，校准误差降低30%，优于纯VLM模型。随着标注比例增加，性能持续改善，主动学习策略显著减少标注成本。多数据集验证显示，VH-CBM在概念准确性和校准方面优于现有方法，尤其在医疗等高风险场景中表现优异。
Applications
该方法适用于医疗影像分析、自动驾驶、工业检测等场景，特别在标注成本高昂或数据隐私敏感的领域。结合少量专家标注与VLM，能快速构建透明、可信的AI系统，满足行业对可解释性和可靠性的需求。
Limitations & Outlook
模型依赖VLM嵌入空间的稳定性，在极端偏域或偏见场景中可能性能下降。高斯过程在大规模数据上计算成本较高，需采用稀疏近似。少量标注虽能提升性能，但在复杂任务中仍需较多专家投入，未来需优化标注策略和模型效率。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多机器（模型），每台机器都能完成特定任务，比如装配或包装。为了让机器更聪明，你可以给它们一些指令（概念），比如“这个零件是红色的”或“这个产品是圆形的”。但有时候，指令不够准确，机器可能会误解。现在，你希望用一种聪明的方法，让工厂里的机器通过少量的示范（专家标注），学习到更准确的指令。你还用一种叫“高斯助手”的智能工具，帮忙分析所有机器的表现，告诉你哪些指令最可靠。这样，即使只给少量示范，工厂里的机器也能更准确地理解任务，工作效率更高，错误更少。这个方法让工厂既能快速适应新任务，又能保持透明和可靠。
ELI14 Explained like you're 14
想象你在学校里学做菜，你有很多食材（模型的知识），但不太清楚每个食材具体怎么用。有时候，你会问老师（专家）：“这个菜需要放盐吗？”老师告诉你一些建议，但不一定每次都完全正确。于是，你用一种聪明的办法，把老师的建议和你自己试验的结果结合起来。你还用一种叫“高斯助手”的聪明工具，帮你分析所有的试验，告诉你哪些建议最靠谱。这样，即使你只听老师几次建议，你也能做出好吃的菜，而且每次都能更接近完美。这种方法让你既能快速学会做菜，又能保证做出来的菜好吃、健康。
Abstract
Concept-bottleneck models (CBMs) are neural classifiers that compute predictions from high-level concepts extracted from the input. CBMs ensure stakeholders can understand the concepts -- and the predictions they entail -- by learning these from concept-level annotations, which are however seldom available. Recent CBM architectures work around this issue by obtaining annotations from Vision-Language Models (VLMs). While greatly broadening applicability, doing so can yield lower quality concepts and therefore less interpretable models. We strike for a middle ground by introducing Vision-plus-Human-guided CBM (VH-CBM), a hybrid approach that exploits both VLMs and a small amount of dense annotations. VH-CBM employs a Gaussian Process in the VLM's embedding space, which captures useful global information about the target domain, to propagate the expert's supervision to any target data point. Our empirical evaluation shows how VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data, while sporting better concept calibration and supporting active learning.