Sampling-based Nyström Approximation and Kernel Quadrature

Kernel methods, such as Gaussian processes and kernel PCA, have revolutionized machine learning by providing flexible, nonlinear modeling capabilities. Early works like Schölkopf et al. (1991) and Scholkopf & Smola (2002) established the theoretical foundation. As datasets grew, the computational cost of kernel matrices became prohibitive, leading to approximations like Nyström (2000) and Random Fourier Features (Rahimi & Recht, 2007). Recent advances include leverage score sampling, DPP-based sampling, and kernel K-means, which aim to improve approximation quality. Despite these efforts, challenges remain in high-dimensional, non-uniform data, where spectral decay assumptions may not hold, and error bounds are less tight. The current state-of-the-art methods often lack rigorous guarantees under realistic sampling schemes, motivating the need for a more comprehensive theoretical framework.

The core issue lies in controlling the approximation error of Nyström methods when landmark points are sampled non-independently or from complex distributions. Traditional bounds assume uniform or leverage-based sampling, which may not reflect practical scenarios like DPP sampling or adaptive selection. Moreover, high-dimensional kernels often exhibit slow spectral decay, making existing error bounds loose or inapplicable. Addressing these limitations requires a unified theoretical approach that accounts for spectral properties, sampling mechanisms, and the underlying measure's geometry. Achieving tight error bounds under these conditions is crucial for reliable large-scale kernel learning, especially in applications demanding high accuracy and robustness.

The paper's key innovations include: 1) integrating statistical learning theory with spectral analysis to derive error bounds under exponential eigenvalue decay; 2) developing Mercer-based low-rank kernel decompositions tailored for non-i.i.d. landmark points, including DPP sampling; 3) extending the analysis to complex sampling schemes, providing theoretical guarantees that surpass traditional uniform sampling bounds. These contributions enable more flexible and accurate kernel approximations, bridging the gap between theory and practice in large-scale settings. The approach also introduces new algorithms for low-rank kernel approximation that are both theoretically sound and computationally feasible.

• �� Decompose the kernel matrix via SVD, analyze spectral decay to derive error bounds under exponential eigenvalue decay assumptions.
• �� Use Tchakaloff’s theorem to convert kernel approximation into finite kernel quadrature, enabling efficient estimation of integral errors.
• �� Incorporate Mercer decomposition of the kernel with respect to the underlying measure, allowing low-rank truncation tailored to landmark points.
• �� Extend spectral analysis to non-i.i.d. landmark points, including DPP sampling, by deriving probabilistic bounds on eigenvalue decay.
• �� Develop subspace selection strategies based on Mercer expansion, optimizing the approximation quality.
• �� Validate theoretical bounds through synthetic and real data experiments, comparing traditional and proposed methods across different sampling schemes.

Experiments involve high-dimensional Gaussian kernels on synthetic datasets and real-world data such as UCI benchmarks. Baselines include standard Nyström, leverage score sampling, and DPP-based methods. Metrics focus on kernel approximation error, integral estimation accuracy, and computational time. Hyperparameters include landmark set size, spectral decay parameters, and sampling schemes. Ablation studies analyze the impact of Mercer truncation and sampling choices. Results demonstrate that the proposed method achieves lower errors (up to 30% improvement) and better scalability, especially in non-uniform and high-dimensional scenarios. The experiments confirm the theoretical error bounds and showcase practical advantages in kernel regression and Bayesian inference tasks.

Theoretical analysis shows that under exponential spectral decay, the expected error diminishes at a rate of O(√(∑_{i>s} σ_i + (log ℓ)^{2d+1}/ℓ)), outperforming classical bounds. Empirical results validate this, with significant error reductions observed in high-dimensional kernel approximation and quadrature tasks. DPP sampling combined with Mercer truncation yields tighter bounds and more robust approximations in non-uniform settings. The experiments also reveal that the method maintains high accuracy with fewer landmark points, demonstrating efficiency and scalability. These findings establish a new benchmark for kernel approximation in large-scale, complex data environments.

This framework benefits large-scale kernel regression, Gaussian process modeling, and Bayesian inference, especially with high-dimensional or non-uniform data. It enables more accurate and computationally feasible kernel approximations, facilitating real-time inference and large dataset analysis. The approach can be integrated into existing machine learning pipelines, improving model robustness and scalability. Additionally, it opens pathways for advanced kernel-based algorithms in scientific computing, signal processing, and data science, where precise integral estimates and spectral approximations are critical.

The spectral decay assumption may not hold for all kernels, particularly in irregular or non-smooth functions, limiting the bounds’ tightness. Computational complexity of Mercer decomposition and DPP sampling remains high for very large datasets, requiring further optimization. Extending the theory to non-positive definite kernels or non-Euclidean spaces presents challenges due to spectral properties. Practical implementation may also face difficulties in accurately estimating spectral decay parameters and eigenfunctions, especially in noisy or high-dimensional data. Future work should focus on scalable algorithms and broader theoretical generalizations.