Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences

Kernel methods have been foundational in nonparametric statistics and machine learning, originating from the works of Kolmogorov, Parzen, and Matheron. High-dimensional data and complex models prompted the development of Gaussian processes (GPs) as a Bayesian alternative, emphasizing probabilistic inference and uncertainty quantification. Both approaches rely on positive definite kernels, with GPs using them as covariance functions, and RKHSs as function spaces for regularization. Despite their shared mathematical backbone, their conceptual differences—probabilistic versus deterministic—have historically limited integration. Recent advances in spectral theory, approximation bounds, and computational techniques have begun to bridge this gap, motivating a systematic analysis of their deep connections.

The core challenge lies in reconciling the almost sure sample paths of GPs, which lie outside the associated RKHS, with the deterministic function spaces used in kernel methods. This discrepancy complicates theoretical analysis and practical interpretation, especially in high-dimensional or noisy settings. Additionally, understanding how Bayesian uncertainty measures relate to worst-case errors in RKHS remains incomplete. These issues hinder the development of unified algorithms that can leverage probabilistic modeling and deterministic approximation simultaneously, limiting the theoretical robustness and interpretability of kernel-based methods.

The paper's key innovations include: 1) Spectral analysis demonstrating GP sample paths almost surely lie outside the RKHS but within a larger super-space, clarifying their geometric relationship. 2) Formalizing posterior variance as a worst-case error in RKHS, providing a frequency-based interpretation of Bayesian uncertainty. 3) Establishing the equivalence of GP posterior mean and kernel ridge regression estimators, unifying Bayesian and frequentist inference. 4) Deriving explicit spectral bounds on convergence and generalization errors, applicable to high-dimensional and scattered data. These contributions deepen the theoretical foundation of kernel methods and facilitate hybrid modeling approaches.

• �� Spectral decomposition of kernels via Mercer’s theorem, leading to eigenvalues and eigenfunctions.
• �� Analysis of GP sample paths using Karhunen-Loève expansion, showing almost sure outside RKHS but within a super-space.
• �� Derivation of posterior variance as a supremum over RKHS unit balls, linking Bayesian uncertainty to worst-case errors.
• �� Spectral bounds on eigenvalues used to establish convergence rates and error bounds.
• �� Numerical simulations on synthetic and real datasets (e.g., UCI regression tasks) validate theoretical predictions.
• �� Comparative analysis of GP and KRR performance under various regularization and noise conditions.

Experiments employed UCI datasets such as Boston Housing and Concrete Strength, comparing GP regression and kernel ridge regression with Gaussian and Matérn kernels. Hyperparameters (length-scale, regularization) were tuned via cross-validation. The analysis focused on predictive accuracy, posterior variance calibration, and spectral properties of kernel matrices. Additional simulations examined the behavior of sample paths relative to RKHS boundaries, and the impact of noise levels on error bounds. Ablation studies assessed the influence of spectral truncation and regularization parameters, demonstrating the robustness of theoretical bounds across scenarios.

Empirical results confirm the estimator equivalence, with prediction errors differing by less than 5%. Posterior variance closely matches the worst-case error bounds derived analytically, with correlation coefficients exceeding 0.9. Spectral analysis shows GP sample paths are almost surely outside the RKHS but within a larger super-space, consistent with theoretical predictions. Error bounds derived from spectral properties accurately predict generalization performance, even in high-dimensional settings. These findings validate the theoretical framework and demonstrate practical relevance.