The data-driven Schroedinger bridge

Probability flow estimation and optimal transport have been foundational in statistics, physics, and machine learning. Schrödinger bridges, originating from 1930s statistical mechanics, seek the most probable evolution between two boundary distributions. Early work by Schrödinger, large deviation theory, and maximum entropy principles laid the groundwork. Recent advances like Sinkhorn algorithms introduced regularized optimal transport solutions, but these are limited by their reliance on explicit densities and low-dimensional discretization. As data complexity and dimensionality increase, traditional methods face computational bottlenecks. This paper addresses these challenges by proposing a sample-based iterative framework that leverages importance sampling and maximum likelihood estimation, enabling scalable high-dimensional probability migration.

The core challenge is to compute the most likely probability evolution between two distributions when only samples are available, and the transition dynamics are known only through simulation or sampling, not explicit formulas. Existing methods depend heavily on discretization and explicit density functions, which become infeasible in high dimensions due to the curse of dimensionality. Moreover, high-dimensional integrals are computationally expensive and numerically unstable. The problem is further complicated by limited sample sizes and the need for robust convergence guarantees. Developing an algorithm that can operate solely on samples, efficiently propagate functions, and ensure convergence in high-dimensional spaces remains an open and pressing issue.

The primary innovation is transforming the classical Schrödinger system into a sample-based iterative algorithm that employs maximum likelihood estimation for boundary densities and importance sampling for propagating functions ϕ and ˆϕ. This approach eliminates the reliance on explicit density functions and grid discretization, making high-dimensional problems tractable. The method integrates the Fortet iterative scheme with modern statistical techniques, ensuring convergence and robustness. Additionally, the algorithm adapts to scenarios with limited samples, providing accurate probability flow estimates without explicit integral evaluation. This fusion of classical theory and modern sampling techniques represents a significant advancement in probabilistic modeling.

• �� Initialize boundary functions ϕ and ˆϕ with samples; • Use importance sampling to propagate ϕ and ˆϕ across iterations; • Estimate marginals via maximum likelihood from samples; • Update boundary functions based on these estimates; • Repeat until convergence. The process involves:
• �� Sampling and resampling to maintain marginal accuracy;
• �� Propagation of functions through importance sampling, avoiding high-dimensional integrals;
• �� Iterative optimization of boundary functions to match observed marginals;
• �� Convergence checks based on residual errors. The framework combines statistical estimation, importance sampling, and iterative optimization, ensuring scalability and stability in high dimensions.

Experiments involved synthetic Gaussian mixtures and high-dimensional probability transfer tasks. In 2D, the method achieved errors below 1.5%, outperforming grid-based methods in speed by orders of magnitude. In higher dimensions (up to 10D), the algorithm maintained stability and captured complex distribution paths. Integral estimates via importance sampling showed over 20% error reduction compared to Monte Carlo. Ablation studies examined sample size effects, iteration counts, and importance sampling parameters, confirming robustness. Comparisons with classical methods demonstrated significant efficiency gains and applicability to large datasets, validating the approach’s scalability and accuracy.

The algorithm achieved sub-1.5% error in 2D Gaussian interpolation, with computational speed improvements of over 50x compared to grid methods. In 10D, it maintained stable convergence, accurately capturing complex probability flows. Variations of importance sampling reduced integral estimation errors by 20%, outperforming standard Monte Carlo. The method demonstrated robustness across different distributions and sample sizes, confirming its suitability for high-dimensional probability transfer tasks.

This approach is immediately applicable to climate modeling, where it estimates atmospheric or oceanic flow trajectories from sparse data. In finance, it enables risk migration analysis with limited historical samples. In biology, it facilitates modeling evolutionary trait distributions over time. The method requires only sample data and known transition dynamics, making it versatile for real-world scenarios. Long-term, integrating neural network parameterizations could further enhance scalability, enabling real-time high-dimensional probability flow estimation in complex systems like weather forecasting, financial markets, and large-scale biological studies.

Dependence on large sample sizes may limit efficiency in extremely high dimensions. Importance sampling efficiency diminishes as dimensionality increases, requiring advanced variance reduction techniques. The method’s performance can degrade with highly skewed or sparse data, and convergence guarantees depend on initializations and sampling quality. Future improvements should focus on adaptive sampling, dimensionality reduction, and hybrid models to address these issues.