Revisiting Active Sequential Prediction-Powered Mean Estimation

The mean estimation problem is a classical task in statistical inference that has garnered renewed interest in the machine learning field. Traditional mean estimation methods typically assume that data is independently and identically distributed, and sufficient samples are available for inference. However, in practical applications, data is often limited, and obtaining true labels can be costly. Therefore, research on how to perform accurate mean estimation under a limited label budget has become an important direction. Recent studies have explored designing efficient mean estimators under various settings and assumptions, such as dealing with adversarial outliers, heavy-tailed distributions, and high-dimensional data.

In the problem of active sequential prediction-powered mean estimation, researchers need to decide whether to query the true label of a sample at each round. If the label is not queried, the prediction from a machine learning model is used instead. Existing methods typically determine query probability by combining an uncertainty-based suggestion with a constant probability, but this strategy may not fully utilize the limited label budget. Additionally, optimizing query probability without relying on current covariates is a significant challenge.

The core innovations of this paper include proposing a query strategy that combines uncertainty suggestion and constant probability, and revealing the impact of constant probability weight on confidence interval width through non-asymptotic analysis. • First, the paper proposes a query strategy that combines uncertainty suggestion and constant probability. This strategy improves estimation accuracy by reducing the influence of the uncertainty component. • Second, it provides a data-dependent confidence interval bound, offering new insights into handling uncertainty components in active learning. • Third, it optimizes query probability through a no-regret learning approach, ensuring convergence to the maximum query probability constraint.

The methodology of this paper includes the following key steps: • Determine query probability by combining uncertainty suggestion and constant probability. Input includes covariates of the sample and uncertainty of model predictions, output is the query probability. • Conduct non-asymptotic analysis to establish a data-dependent confidence interval bound. • Use a no-regret learning approach to optimize query probability, ensuring convergence to the maximum query probability constraint. • Validate the method through experiments on multiple datasets, analyzing the impact of different mixing parameters on confidence interval width.

The experimental design includes validating the proposed method on three real-world datasets and one synthetic dataset. Baselines used include existing mixed strategies and uniform sampling strategies. Experimental metrics include confidence interval width and coverage. Key hyperparameters include the choice of mixing parameters and the setting of the label budget. Ablation studies analyze the impact of different strategies on the results.

Experimental results show that the proposed method outperforms baselines on multiple datasets. Specifically, the smallest confidence width occurs when the constant probability weight is near one. Additionally, the proposed method optimizes query probability through a no-regret learning strategy without relying on current covariates, demonstrating its effectiveness. The experiments also indicate that reducing the influence of the uncertainty component can improve estimation accuracy.

This method has potential applications in various scenarios. • In medical diagnosis, it can improve diagnostic accuracy under a limited label budget, assisting doctors in making quicker decisions. • In financial forecasting, it can conduct more accurate market analysis with limited data, helping investors make more informed decisions. • In autonomous driving, it can optimize the use of sensor data, enhancing vehicle decision-making capabilities and driving safety.

Despite the excellent performance of this method on multiple datasets, there are still some limitations. • The method may lead to amplified estimator variance when the uncertainty predictor is inaccurate, affecting the reliability of results. • On some datasets, the constant probability strategy may not fully utilize the model's uncertainty information. • The performance of the method depends on selecting an appropriate mixing parameter, which may require additional tuning work. Future research could explore dynamically adjusting the mixing parameter to adapt to different dataset characteristics without increasing computational complexity.