Semi-Supervised Learning with Noisy Proxy Covariates: Generalization Bounds and Distribution Regression

Recent advances in pretraining, such as BERT and CLIP, have revolutionized feature extraction, enabling models to learn rich representations from unlabeled data. Despite success, these features often contain noise due to domain mismatch, measurement errors, or imperfect training data. Traditional semi-supervised methods like graph-based regularization or manifold assumptions rely on clean data structures, which are compromised by noisy proxies. Distribution regression, where each covariate is a distribution estimated from finite samples, exemplifies the challenge of noisy features. Existing work has shown that spectral methods and kernel embeddings can capture complex data geometry, but their robustness under proxy noise remains underexplored. Addressing this gap is crucial for deploying models in real-world noisy environments.

The core challenge is how to effectively utilize large amounts of noisy proxy covariates to improve regression accuracy with limited labels. Proxy variables, derived from pretrained models or measurement processes, introduce bias and variance, distorting the spectral structure of the data. Existing spectral algorithms are sensitive to noise, leading to inaccurate eigenspaces and degraded generalization. The problem is compounded in distribution regression, where each covariate is a distribution estimated from finite samples, further amplifying measurement errors. Developing a theoretical framework that explicitly accounts for proxy noise and quantifies its impact on spectral estimation and generalization is essential to overcoming these limitations.

This paper introduces a spectral kernel framework that explicitly models proxy noise and leverages unlabeled data to estimate the eigenspaces of an integral operator. Key innovations include: 1) deriving finite-sample bounds that incorporate proxy perturbation effects, 2) unifying distribution regression as a special case within the spectral learning paradigm, 3) establishing conditions under which unlabeled proxies can eliminate the statistical cost of spectral estimation, and 4) providing practical algorithms with validation-based parameter tuning and guarantees for approximate eigendecomposition. These contributions significantly extend the theoretical understanding of spectral methods under measurement errors, enabling robust semi-supervised learning in noisy environments.

• �� Construct the kernel integral operator L from all proxy covariates, estimating its spectral structure via eigen-decomposition of the kernel matrix.
• �� Select the top s eigenfunctions to form a low-dimensional feature map, “Φ”, capturing the dominant data geometry.
• �� Use the eigenfunctions to project both labeled and unlabeled proxies, ensuring spectral stability through Davis–Kahan bounds.
• �� Fit a ridge regression model on labeled data within this spectral subspace, controlling bias and variance via regularization parameter ξ.
• �� Derive finite-sample excess risk bounds by analyzing the combined errors from spectral approximation, proxy noise, and label estimation, explicitly relating these to sample sizes and spectral decay.
• �� Extend the framework to distribution regression by viewing each distribution as a proxy, with the proxy noise controlled by sample size m.
• �� Implement validation-based tuning for spectral dimension s and regularization ξ, ensuring practical robustness.
• �� Incorporate approximate eigendecomposition methods, analyzing their impact on statistical guarantees.

Experiments involve synthetic datasets with controlled proxy noise levels and real-world tasks such as image classification with pretrained features. Baselines include supervised ridge regression, classical semi-supervised methods, and spectral algorithms ignoring noise. Metrics focus on mean squared error and prediction accuracy. Hyperparameters like spectral dimension s and regularization ξ are tuned via validation. The study varies proxy noise levels, unlabeled sample size N, and distributional complexity (e.g., bag size m in distribution regression). Results show that the proposed spectral approach consistently outperforms baselines in low-label regimes, with errors approaching the theoretical n^{-rac{q}{q+1}} rate. Ablation studies confirm the importance of spectral truncation and proxy quality.

Quantitative analysis reveals a 15% error reduction over supervised methods at 10% labels, with errors approaching the optimal n^{-rac{q}{q+1}} rate when proxy noise is controlled and N is large. Distribution regression experiments demonstrate that increasing bag size m reduces proxy noise, leading to improved accuracy and convergence to the oracle rate. Theoretical bounds match empirical trends, confirming that unlabeled proxies can effectively eliminate the spectral estimation cost under suitable conditions. Validation-based parameter tuning maintains these rates in practice, with robustness to approximate eigendecomposition, ensuring scalability.

This method applies to scenarios where pretrained models generate noisy features, such as NLP embeddings, visual representations, or biological measurements. It benefits tasks with scarce labels but abundant unlabeled data, including medical diagnosis, financial forecasting, and recommendation systems. The spectral framework is suitable for high-dimensional, distributed data environments, providing a robust way to leverage large-scale unlabeled proxies. Its ability to quantify proxy noise impact makes it valuable for real-world deployment, where measurement errors are inevitable.

The approach assumes proxy noise satisfies certain regularity and boundedness conditions; deviations from these assumptions may reduce effectiveness. Computational complexity of kernel eigen-decomposition limits scalability, though randomized methods can mitigate this. The spectral bounds depend on eigenvalue decay and eigengap, which may not hold in highly irregular data distributions. Additionally, the method's performance deteriorates if proxy noise dominates or proxy data are insufficiently abundant. Future work should focus on adaptive spectral truncation, scalable algorithms, and relaxing noise assumptions for broader applicability.