Decision-Making with Auto-Encoding Variational Bayes

Bayesian inference has become central in machine learning, enabling principled uncertainty quantification. Variational autoencoders (VAEs) introduced a scalable framework for high-dimensional data, optimizing the Evidence Lower Bound (ELBO). Despite success in representation learning and generation, ELBO-based training often produces biased posteriors, limiting decision accuracy. Recent advances like IWAE, EP, and CHIVI aim to improve posterior approximation, but challenges remain in balancing bias and variance, especially in complex models such as single-cell RNA sequencing and multi-hypothesis testing. These applications demand accurate posterior estimates for reliable decision-making. The limitations of relying solely on ELBO motivate exploring alternative objectives and fusion strategies to enhance posterior fidelity and decision robustness.

The core issue is that models trained via ELBO tend to underestimate posterior variance and do not always produce accurate approximations for decision tasks. This bias affects downstream inference, leading to suboptimal decisions, especially in high-stakes scenarios like genomics and medical diagnosis. Existing importance sampling approaches often suffer from high variance or bias when the proposal distribution is close but not identical to the true posterior. Furthermore, model selection based on ELBO may favor models that fit the data well but do not produce the best posterior for decision-making. Addressing these issues requires a framework that can generate diverse, high-quality approximate posteriors and fuse them effectively for robust inference.

The paper introduces a multi-objective variational training scheme that produces multiple approximate posteriors by optimizing different objectives (e.g., IWELBO, χ-VAE). These diverse approximations are then combined using multiple importance sampling (MIS), which adaptively weights each proposal based on its relevance. Theoretically, the authors show that intentionally deviating from the variational distribution can improve importance sampling efficiency, especially when the proposal better captures complex posterior features. This fusion strategy reduces bias and variance, leading to more accurate posterior expectations and decision metrics. The approach is validated in linear models like pPCA and extended to real-world tasks such as single-cell RNA-seq analysis and multi-hypothesis testing, demonstrating broad applicability.

• �� Train multiple models (VAE, IWAE, χ-VAE) with different objectives to generate diverse approximate posteriors. • Use these models to fit multiple variational distributions, each capturing different aspects of the true posterior. • Apply theoretical bounds based on spectral norms to assess importance sampling error and guide proposal selection. • Implement multiple importance sampling (MIS), combining samples from all proposals with adaptive weights derived from importance weights. • Evaluate posterior expectations and decision metrics, such as FDR in genomics or classification accuracy in MNIST. • Conduct experiments on synthetic pPCA data and real single-cell RNA-seq datasets, comparing bias, variance, and decision performance. • Use metrics like MAE, PSIS diagnostics, and FDR control to validate improvements over traditional ELBO-based methods.

Experiments involve synthetic pPCA data and real single-cell RNA-seq datasets. The models trained include VAE, IWAE, and χ-VAE, with hyperparameters tuned for particle number and model complexity. Posterior expectation accuracy is measured via MAE, while importance sampling diagnostics (PSIS) evaluate proposal quality. The fusion method (IWAE-MIS) is compared against single-model approaches, showing significant reductions in bias and variance. In classification tasks on MNIST, the approach improves decision metrics like AUPRC and FDR control. The experiments confirm that combining multiple approximate posteriors yields more reliable and robust decision-making, especially in high-dimensional, complex models.

In simulations, χ-VAE and IWAE achieved MAEs of 0.030 and 0.032, outperforming the VAE (0.103). On real data, the three-step approach reduced FDR estimation error to 0.27, surpassing baseline methods. The fusion of multiple models via MIS improved importance sampling efficiency, decreasing bias and variance. Theoretical bounds based on spectral norms validated that deviating from the variational distribution enhances sampling performance. Across all scenarios, the proposed framework consistently outperformed traditional ELBO-based methods, demonstrating its effectiveness in complex decision tasks.

This framework is applicable in genomics for differential gene expression analysis, in medicine for diagnostic decision-making, and in AI for reinforcement learning. It enables more accurate uncertainty quantification and robust decision-making in high-dimensional models, especially where traditional variational methods fall short. The approach can improve the reliability of scientific discoveries, clinical diagnoses, and autonomous systems by providing better posterior estimates and decision metrics.

Theoretical analysis is primarily limited to linear models like pPCA; extending guarantees to deep nonlinear models remains challenging. Computational costs increase with multiple models and sampling steps, limiting scalability. Model selection and hyperparameter tuning require expertise, affecting robustness. Further work is needed to automate these processes and reduce computational overhead, especially for large-scale applications.