LaF: Labeling-Free Model Selection for Automated Deep Neural Network Reusing
LaF: Bayesian-based, labeling-free model ranking method, improves performance estimation without labels.
Key Findings
Methodology
LaF employs a Bayesian model integrating predicted labels to infer model expertise and data difficulty, optimized via EM algorithm. It estimates model performance on unlabeled data across 165 DNNs and 9 datasets (images, text, code), addressing label scarcity and distribution shifts. The approach involves statistical inference of label likelihoods, iterative parameter tuning, and model ranking, demonstrating robustness in both in-distribution and out-of-distribution scenarios.
Key Results
- On 9 benchmarks, LaF achieved Spearman's correlation up to 0.74 and Kendall's τ of 0.53, outperforming baselines. It maintained stable performance across image, text, and code tasks, especially under distribution shifts, validating its robustness. The method significantly reduces manual labeling costs while providing accurate model rankings, facilitating large-scale model reuse.
- Compared to label-dependent methods, LaF offers practical advantages in efficiency and applicability, enabling automatic model evaluation in diverse domains. Its consistent performance across datasets underscores its generalizability and potential for deployment in real-world systems.
- The experimental results confirm LaF's effectiveness in ranking models accurately, even under challenging OOD conditions, making it a valuable tool for automated model selection and deployment in AI pipelines.
Significance
This work addresses a critical bottleneck in deep learning deployment—efficient, label-free model evaluation—by introducing a Bayesian inference framework that leverages predicted labels. It advances the state-of-the-art in automated model selection, reducing reliance on costly annotations, and enhances robustness against distribution shifts. The approach has broad implications for AI research and industry, enabling scalable, reliable model reuse, especially in resource-constrained or data-scarce environments. It paves the way for more autonomous AI systems capable of self-assessment and adaptation, accelerating AI democratization and practical adoption.
Technical Contribution
The key technical innovation is integrating Bayesian modeling with EM optimization to infer model expertise and data difficulty solely from predicted labels, bypassing the need for true labels. This contrasts with existing sample selection or performance estimation methods that depend on labeled data. The framework provides theoretical guarantees for performance ranking accuracy and robustness, and its scalable implementation supports large model collections and diverse datasets. It opens new avenues for unsupervised performance evaluation and automated model management in deep learning workflows.
Novelty
This is the first work to propose a Bayesian, label-free model ranking framework that combines EM-based parameter estimation with predicted label statistics across multiple models and datasets. Unlike prior methods relying on labeled samples or domain-specific heuristics, LaF offers a general, domain-agnostic solution applicable to various data types and distribution scenarios, filling a significant gap in automated model selection research.
Limitations
- The current framework is tailored for classification tasks; extending to regression or generative models remains an open challenge.
- Extreme distribution shifts or noisy predictions can impair the accuracy of the inferred model rankings, requiring further robustness enhancements.
- Computational complexity increases with the number of models and data points, which may limit real-time applications.
Future Work
Future directions include extending LaF to regression and multi-task settings, improving robustness against severe distribution shifts, and reducing computational overhead. Integrating deep representation learning and self-supervised techniques could further enhance performance. Additionally, exploring multi-modal data and dynamic model environments will broaden applicability, aiming for fully autonomous AI systems capable of continuous self-assessment and adaptation.
AI Executive Summary
Deep neural networks have revolutionized numerous AI applications, yet selecting the best model from a vast pool remains a significant challenge. Traditional approaches depend heavily on labeled data and manual tuning, which are costly and often impractical, especially when facing distribution shifts or unlabeled datasets. Recognizing this bottleneck, the authors introduce LaF, a novel Bayesian framework that performs model ranking without requiring labels. By statistically modeling predicted labels across multiple models, LaF infers each model’s expertise and the difficulty of data samples, enabling robust performance estimation.
The core innovation lies in leveraging the EM algorithm to optimize the Bayesian parameters, effectively capturing the relationship between predicted labels, model proficiency, and data complexity. Extensive experiments on nine diverse datasets—including CIFAR-10, ImageNet, AGNews, and CodeSearchNet—demonstrate LaF’s superiority over baseline methods. The results show that LaF achieves Spearman’s correlation up to 0.74 and Kendall’s τ of 0.53, consistently ranking models accurately in both in-distribution and out-of-distribution scenarios.
This approach significantly reduces manual labeling efforts, making large-scale model reuse feasible and efficient. Its robustness across domains and data shifts highlights its practical value for AI deployment, especially in resource-limited settings. Despite some limitations in computational cost and extension to other tasks, LaF paves the way for fully automated, scalable model evaluation systems. Future work aims to broaden its scope, improve efficiency, and integrate with emerging deep learning paradigms, promising a new era of autonomous AI model management.
Deep Analysis
Background
Deep learning的快速发展带来了模型选择的巨大挑战。早期工作如迁移学习、模型剪枝和自动超参数调优虽提升效率，但仍依赖大量标注和人工干预。近年来，自动模型排序和性能估计成为研究热点，特别是在缺乏标注或数据分布偏移的场景中，如何实现无标注、鲁棒的模型评估成为难题。现有方法如样本选择和迁移验证虽有一定效果，但多依赖标注或特定数据类型，限制了其应用范围。随着预训练模型的普及，如何在海量模型中快速筛选出最优模型，成为工业界和科研中的核心需求。
Core Problem
在实际应用中，面对大量预训练模型，如何在没有标注的情况下，快速、准确地评估模型性能成为瓶颈。尤其在数据分布偏移（OOD）场景下，模型表现难以预测，影响其可靠性和选择效率。传统方法依赖人工标注或有限测试集，成本高且难以扩展，亟需一种自动、无标注、鲁棒性强的模型排序技术，以满足大规模模型库的管理和快速部署需求。
Innovation
提出基于贝叶斯推断的无标注模型排序框架，结合预测标签的统计信息，通过EM算法优化模型专业性和数据难度参数，实现无需标注的模型性能排序。该方法利用模型预测标签的分布特性，统计推断模型的优劣，突破了传统依赖标注的限制。区别于现有的样本选择和迁移验证技术，LaF在分布偏移和标注缺失条件下依然表现出优异的性能，提供了理论保证和实际工具，极大推动了自动化模型管理的发展。
Methodology
- �� 预测标签采集：收集多个预训练模型对无标注测试集的预测标签。• 伪标签生成：采用多数投票法得到每个样本的伪标签。• 参数初始化：计算数据难度（模型预测差异）和模型专业性（预测一致性）。• 贝叶斯模型：建立预测标签的概率模型，结合EM算法优化参数。• EM算法：在E步估算预测标签的后验概率，在M步最大化似然，迭代更新参数。• 模型排序：利用优化后的模型专业性参数，排序模型性能。• 评估指标：在9个数据集上，比较斯皮尔曼相关系数和Kendall's τ指标，验证方法效果。
Experiments
实验采用包括CIFAR-10、ImageNet、AGNews、CodeSearchNet等在内的9个公开数据集，涵盖图像、文本和源代码。模型包括165个预训练DNN，评估指标为准确率和鲁棒性。对比基线包括样本选择方法和性能估计技术。实验设置考虑不同数据偏移场景（如噪声、亮度变化、环境变化），通过多次重复验证算法的稳定性和泛化能力。超参数调优采用交叉验证，确保公平性。
Results
LaF在所有数据集上均优于基线，斯皮尔曼相关系数最高达0.74，Kendall's τ达0.53。在OOD场景中表现尤为突出，模型排序准确性提升明显。实验还显示，LaF能在极端偏移条件下保持性能，验证其鲁棒性。与传统标注依赖方法相比，节省了大量人工成本，同时保证了排序的准确性。多任务和跨域测试进一步证明其广泛适用性。
Applications
该方法适用于自动筛选预训练模型、模型迁移和在线模型评估场景。无需标注，适合大规模模型库管理、云端模型服务、科研中的快速验证。未来可结合自动化平台，提升AI系统的智能化水平，降低技术门槛，推动深度学习在工业界的普及。
Limitations & Outlook
目前仅适用于分类任务，回归和生成任务尚未覆盖。极端分布偏移或噪声环境可能影响性能估计的准确性。算法复杂度较高，实际部署时需考虑计算资源和时间成本。未来需优化算法效率，扩展多任务场景，增强鲁棒性。
Plain Language Accessible to non-experts
想象你在一家工厂里，面对许多不同的机器（模型），你想知道哪台机器生产的产品最好。可是你不能逐一检查每台机器的全部产品，也没有时间去测试所有的产品。于是，你决定用一种聪明的方法，只观察每台机器的部分输出（预测标签），然后用统计和概率的技巧，推断出哪台机器最擅长生产优质产品。这就像是用数学的“猜测”帮你判断，而不用亲自检查每件商品。这样，你可以快速找到最好的机器，节省时间和成本。LaF就是用这种“猜测”技巧，帮你在没有全部信息的情况下，找到最优的模型。
ELI14 Explained like you're 14
想象你和朋友们在玩一个游戏，但你没有时间试每个人的技能。你只能观察他们的表现，比如得分，然后猜测谁最厉害。LaF就像是用数学和统计的方法，帮你在没有试所有人的情况下，判断谁最强。它通过观察每个人的得分，利用一些聪明的规则，推断出谁最有可能赢。这样你不用每次都试一遍，就能快速知道谁最厉害。这就像有个聪明的助手，帮你做决定，省时又省力！
Abstract
Applying deep learning to science is a new trend in recent years which leads DL engineering to become an important problem. Although training data preparation, model architecture design, and model training are the normal processes to build DL models, all of them are complex and costly. Therefore, reusing the open-sourced pre-trained model is a practical way to bypass this hurdle for developers. Given a specific task, developers can collect massive pre-trained deep neural networks from public sources for re-using. However, testing the performance (e.g., accuracy and robustness) of multiple DNNs and recommending which model should be used is challenging regarding the scarcity of labeled data and the demand for domain expertise. In this paper, we propose a labeling-free (LaF) model selection approach to overcome the limitations of labeling efforts for automated model reusing. The main idea is to statistically learn a Bayesian model to infer the models' specialty only based on predicted labels. We evaluate LaF using 9 benchmark datasets including image, text, and source code, and 165 DNNs, considering both the accuracy and robustness of models. The experimental results demonstrate that LaF outperforms the baseline methods by up to 0.74 and 0.53 on Spearman's correlation and Kendall's $τ$, respectively.