BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learning

Active learning (AL) is a machine learning technique aimed at improving model performance while reducing annotation costs by selectively labeling data. With the development of foundation models, identifying valuable instances has become easier. However, existing selection strategies lack robustness across different models, annotation budgets, and datasets. In recent years, researchers have attempted to address this issue by developing new selection strategies, but these often rely on performance-related heuristics, which may perform poorly in some scenarios. To better evaluate the effectiveness of existing strategies, oracle strategies have been introduced as a reference point. These strategies approximate optimal selection by accessing ground-truth information unavailable in practical AL scenarios. However, current oracle strategies face challenges in scaling to large datasets and complex deep neural networks.

Existing active learning strategies lack robustness across different models, annotation budgets, and datasets, making it difficult to achieve optimal performance in large-scale datasets and complex deep neural networks. This problem is significant because, as dataset sizes continue to grow, annotation costs are increasing, necessitating a strategy that can work efficiently in large-scale scenarios. Furthermore, existing strategies are typically fixed throughout the AL process, limiting their ability to adapt to distribution shifts caused by iteratively annotating new instances.

BoSS's core innovation lies in its scalability and flexibility. First, BoSS constructs a set of candidate batches through an ensemble of selection strategies and selects the batch yielding the highest performance gain. Second, BoSS achieves efficient performance evaluation by freezing the pretrained backbone and retraining only the final layer during selection. This approach not only reduces computational costs but also enhances the stability of performance evaluation. Additionally, BoSS can be easily extended with new strategies as they emerge, ensuring it remains a reliable oracle strategy in the future.

• �� BoSS first constructs a diverse pool of candidate batches through an ensemble of selection strategies. • It then adopts a performance-based perspective, selecting the candidate batch that, once annotated, leads to the highest performance improvement. • For efficiency, BoSS freezes the pretrained backbone and assesses the performance improvement of candidate batches by retraining only the final layer during selection. • By combining an ensemble-based preselection of candidate batches, a performance-based batch assessment, and a frozen backbone, BoSS serves as a batch oracle strategy that also works in large-scale deep AL settings.

The experimental design includes evaluating BoSS's performance on multiple image datasets, including large-scale datasets like ImageNet. Pretrained Vision Transformers (ViTs) are used as models, and 20 active learning cycles are conducted. Baselines include existing oracle strategies such as CDO and SAS, as well as state-of-the-art active learning strategies. Evaluation metrics include accuracy improvement and computational cost.

Experimental results show that BoSS outperforms existing oracle strategies under comparable computational constraints, especially on large-scale datasets. On ImageNet, BoSS achieves a performance improvement of over 15% compared to random sampling. Additionally, current state-of-the-art active learning strategies still fall noticeably short of oracle performance, especially in large-scale multiclass datasets, indicating potential for developing stronger strategies.

BoSS's excellent performance on large-scale datasets makes it suitable for large-scale image classification tasks that require efficient annotation. It can be applied in fields such as autonomous driving and medical image analysis, helping to reduce annotation costs and improve model performance. Furthermore, BoSS's flexibility allows it to adapt to different datasets and model architectures, offering broad industrial application prospects.

BoSS relies on existing selection strategies for candidate batch selection, which may impact overall performance if the strategies are not well-developed. While BoSS excels on large-scale datasets, its advantages on small-scale datasets have not been fully validated. Additionally, the implementation of BoSS requires certain computational resources, which may not be suitable for resource-constrained environments. Future research could focus on further optimizing the computational efficiency of BoSS, particularly in resource-constrained environments.