Optimal Rates for Regularized Conditional Mean Embedding Learning

Kernel methods have become a cornerstone in nonparametric statistics, enabling flexible modeling of complex data distributions. The conditional mean embedding (CME) extends this by representing conditional distributions as elements in a reproducing kernel Hilbert space (RKHS). Early work by Fukumizu et al. introduced the operator-theoretic definition, but challenges remained in analyzing convergence rates, especially in infinite-dimensional spaces and under model misspecification. Recent advances incorporated spectral and interpolation techniques, yet many relied on restrictive assumptions such as target space finite-dimensionality or strong smoothness conditions. These limitations hindered the application of CME in real-world, high-dimensional problems where models are often misspecified, and the true conditional distribution lies outside the assumed RKHS.

The core problem addressed is establishing the optimal convergence rate of empirical CME estimators under misspecification, where the true conditional mean operator does not belong to the assumed function space. Existing results either assume finite-dimensional target spaces or impose strong spectral decay conditions, which are not always valid. Moreover, there is a lack of theoretical lower bounds matching the upper bounds in infinite-dimensional settings. This gap limits the understanding of the fundamental limits of CME estimation, restricting its use in practical high-dimensional inference tasks, such as causal discovery and probabilistic modeling in complex systems.

The paper introduces a novel approach by constructing a vector-valued interpolation space via tensor products, bridging the operator space of Hilbert-Schmidt operators with a function space framework. This allows the analysis of CME estimation in more general, possibly infinite-dimensional, and misspecified settings. The key innovations include: 1) defining the space [G]α that captures the regularity of the target CME; 2) leveraging spectral properties of covariance operators to derive convergence rates; 3) proving the rate $O(\log n / n)$ is optimal by establishing a matching lower bound. This framework extends the theoretical understanding of kernel embedding methods, making them applicable to broader classes of problems without restrictive assumptions.

• �� Construct the vector-valued interpolation space [G]α using spectral decomposition of the covariance operator CXX, linking eigenvalues and eigenfunctions to the space’s structure.
• �� Map the Hilbert-Schmidt operator space S2(HX, HY) into the function space G via the isomorphism Ψ, then embed into the space of functions with regularity α.
• �� Derive the convergence rate by analyzing the spectral decay of covariance operators, applying concentration inequalities and spectral bounds.
• �� Establish the lower bound by constructing worst-case scenarios, demonstrating that the $O(\log n / n)$ rate cannot be improved in the general setting.
• �� Show that the empirical estimator converges within the interpolation space norm, with adaptivity to unknown smoothness levels.

Simulations with synthetic data generated from known Gaussian and Matérn kernels tested the convergence of the proposed estimator under various spectral decay conditions. Real datasets from causal inference tasks validated the robustness and practical performance. Hyperparameters such as regularization λ and spectral cutoff were tuned via cross-validation. Ablation studies compared the effect of different spectral decay assumptions, confirming the theoretical predictions. Results consistently showed the estimator attains the predicted $O(\log n / n)$ rate, outperforming classical methods that rely on restrictive assumptions.

The main result confirms that in the misspecified setting, the kernel ridge regression estimator of CME achieves an optimal $O(\log n / n)$ convergence rate in the interpolation norm. This rate holds without assuming the target operator is finite-rank or highly smooth, broadening the scope of kernel methods. The spectral analysis validates the rate's dependence on eigenvalue decay, and the lower bound construction proves its optimality. Empirical tests demonstrate the estimator’s superior convergence speed and robustness across various spectral regimes, confirming the theoretical findings and practical relevance.

This framework enables high-precision causal inference, probabilistic modeling, and Bayesian inference in high-dimensional, complex environments. It is particularly suited for scenarios where the true conditional distribution is unknown or lies outside the assumed RKHS, such as in genomics, neuroscience, and economics. The method requires only mild spectral decay conditions, making it adaptable to various kernels and data types. Its ability to handle misspecification enhances its utility in real-world applications where model assumptions are often violated.

The approach relies on spectral decay assumptions that may not hold for highly irregular kernels or data with slow eigenvalue decay. Computational complexity of spectral decomposition limits scalability to very large datasets. Extending the theory to nonlinear or non-stationary environments remains challenging. Future work must address these issues to enable broader practical deployment, especially in real-time or resource-constrained settings.