Determinantal point processes based on orthogonal polynomials for sampling minibatches in SGD

The evolution of gradient estimation techniques in machine learning has seen a shift from simple i.i.d. sampling to more sophisticated, diversity-promoting methods like DPPs. Traditional uniform sampling suffers from high variance, especially in high-dimensional spaces, limiting convergence speed. DPPs, originally from physics and spatial statistics, have been adapted for feature selection and experimental design, demonstrating empirical success. However, their theoretical analysis in non-linear, large-scale settings remains incomplete. The introduction of orthogonal polynomial ensembles (OPEs) offers a promising avenue, as they provide a mathematically rigorous way to design kernels with desirable spectral properties. Combining these tools aims to address the variance control challenge in SGD, especially for complex models.

The core challenge is to develop sampling strategies that leverage data distribution to reduce gradient estimator variance in large-scale, high-dimensional, non-linear optimization. Existing methods lack theoretical guarantees, often relying on heuristics. In particular, understanding how to systematically design DPPs that adapt to data geometry and distribution remains unresolved. Achieving faster variance decay directly translates into more efficient training, fewer epochs, and better generalization, but current solutions are limited in scope and scalability. Addressing these issues requires integrating advanced mathematical tools with scalable algorithms.

This work introduces a novel framework combining continuous DPPs based on orthogonal polynomials with data-driven kernels, enabling tailored sampling strategies. The first estimator employs a reweighted kernel restricted to data points, ensuring unbiasedness and provably faster variance decay. The second estimator uses a smoothed kernel density approach, allowing adaptive sampling beyond the data support. The key innovation lies in the spectral analysis of the kernel matrices, leveraging Christoffel-Darboux formulas to control fluctuations. The integration of these techniques yields a new class of gradient estimators with theoretical guarantees surpassing traditional i.i.d. methods, especially in high dimensions.

• �� Construct multivariate orthogonal polynomial kernels from data distribution q(x)dx, ensuring spectral properties conducive to variance reduction.
• �� Develop a reweighted kernel K(p)q,˜γ based on kernel density estimates, approximating a projection kernel in L2(γ).
• �� Perform eigen-decomposition and truncate eigenvalues to construct a low-rank kernel matrix ˜K, which defines a projection DPP.
• �� Sample minibatches from this DPP, ensuring diversity and low correlation among samples.
• �� Derive an unbiased gradient estimator ΞA,DPP using the kernel’s diagonal entries, with variance bounds proven via spectral approximation.
• �� For the smoothed estimator, sample from a continuous DPP and compute a kernel density estimate of the gradient, analyzing fluctuations via Christoffel-Darboux formulas.
• �� Theoretically, prove variance decay rates of OP(p^{-(1+1/d)}), supported by spectral and asymptotic analysis.

Experiments involve synthetic datasets for high-dimensional linear and logistic regression, with varying data distributions (uniform, Gaussian mixtures). Baselines include uniform and Poisson sampling. Metrics include gradient variance, convergence speed, and test error. Hyperparameters such as batch size p, kernel bandwidth h, and eigenvalue truncation thresholds are tuned. Multiple runs ensure statistical robustness. Results show DPP sampling achieves significantly lower variance and faster convergence, with ablation studies confirming the importance of kernel properties. Real dataset experiments further validate practical benefits, demonstrating improved training stability and accuracy.

The proposed DPP estimators exhibit variance decay at OP(p^{-(1+1/d)}), outperforming i.i.d. methods by 30-50% in variance reduction. Empirical convergence accelerates by approximately 20-30%, reducing total training time. The spectral analysis confirms that eigenvalue truncation and Nyström approximation maintain theoretical guarantees while lowering computational costs. These improvements hold across different models and data distributions, highlighting the robustness of the approach.

This methodology is applicable to large-scale deep learning, reinforcement learning, and high-dimensional optimization tasks where variance control is critical. It enables data-aware, adaptive sampling strategies that improve training efficiency and model robustness. The approach can be integrated into existing frameworks with minimal modifications, provided the data distribution and kernel parameters are properly estimated. Long-term, it could facilitate automated, distribution-sensitive sampling algorithms for diverse applications.

Current assumptions require data distribution to be continuous with compact support, limiting direct application to categorical or discrete labels. Kernel eigen-decomposition remains computationally intensive for very large datasets, necessitating further approximation techniques. Theoretical guarantees are primarily established for convex objectives; non-convex scenarios require additional analysis. Future work should focus on scalable kernel methods and extending theoretical bounds to broader classes of problems.