Conditioning by adaptive sampling for robust design
Proposes Conditioning by Adaptive Sampling (CbAS) for protein design, leveraging generative models to estimate conditional distributions efficiently.
Key Findings
Methodology
CbAS employs model-based adaptive importance sampling to approximate the input distribution conditioned on desired properties, avoiding direct optimization of predictive models. It integrates variational autoencoders (VAE) and importance sampling to iteratively refine the distribution towards target property events. The approach is model-agnostic, requiring no differentiability of the oracle, making it suitable for black-box predictors. The core mechanism involves minimizing KL divergence between the target conditional and the current model, ensuring convergence to the desired distribution even for rare events.
Key Results
- In protein fluorescence optimization, CbAS achieved approximately 15% higher maximum fluorescence compared to methods like Gomez-Bombarelli et al. (2018) and Gupta & Zou (2019). It generated sequences with higher property scores and fewer unrealistic proteins. The method reduced the number of samples needed by over 50% while maintaining high-quality outputs. Ablation studies confirmed the importance of importance sampling and iterative conditioning in performance gains.
- In molecular property optimization, CbAS outperformed baseline methods in efficiency and accuracy, demonstrating robustness across different property thresholds. It successfully identified high-value sequences in complex, high-dimensional spaces, validating its applicability in real-world molecular design tasks.
- The experiments showed that the iterative importance sampling approach effectively handles rare event conditioning, with the KL divergence between the estimated and true conditional distributions decreasing steadily over iterations, indicating convergence.
Significance
This work addresses a fundamental challenge in molecular and protein design—how to efficiently sample in high-dimensional, discrete spaces conditioned on rare events. Traditional Bayesian optimization struggles with sparse target regions and non-differentiable predictors. CbAS offers a theoretically grounded, flexible solution that can handle black-box models and complex constraints, opening new avenues for accelerated discovery in drug development, enzyme engineering, and materials science. Its ability to robustly approximate target distributions enhances the reliability and scalability of computational design workflows, bridging the gap between predictive modeling and experimental validation.
Technical Contribution
The primary technical innovation lies in formulating the conditional distribution estimation as a KL divergence minimization problem, solved via iterative importance sampling. This framework generalizes existing model-based optimization algorithms like EDA and IGO, incorporating importance weights to handle rare events. The method's flexibility allows it to work with various generative models, including VAEs and latent-variable models, even with intractable likelihoods. Theoretical guarantees of convergence and stability are provided, and the approach extends naturally to multi-objective and specification-based constraints, broadening its applicability.
Novelty
This is the first systematic development of a model-based, importance sampling-driven framework for high-dimensional, discrete, and sparse-event conditioning in molecular design. Unlike gradient-based or GAN-based methods, CbAS does not require differentiability and can operate with black-box predictors. Its iterative, importance-weighted approach effectively overcomes the challenge of rare event estimation, setting a new standard for robustness and efficiency in conditional generative modeling for scientific applications.
Limitations
- The method still requires substantial computational resources, especially for very high-dimensional spaces or extremely rare events, due to the need for large sample sizes.
- Dependence on the quality of the generative model; biased or poorly trained models can impair performance.
- Handling multiple conflicting constraints or highly multimodal target distributions remains challenging, requiring further methodological enhancements.
Future Work
Future directions include extending CbAS to multi-objective optimization, integrating reinforcement learning for adaptive exploration, and developing more efficient sampling strategies. Additionally, applying the framework to other domains like materials discovery and complex biological systems will test its generality. Improving scalability and reducing computational costs are ongoing goals, alongside theoretical analysis of convergence rates in more complex settings.
AI Executive Summary
Designing molecules and proteins with desired properties has long been a central challenge in chemistry and bioengineering. Traditional experimental approaches, such as directed evolution, are costly and time-consuming, relying heavily on trial-and-error. Recent advances in machine learning have begun to transform this landscape, enabling the use of predictive models—so-called oracles—to guide design. However, directly optimizing these models often leads to unreliable results, especially when the inputs deviate from training data, due to model pathologies and the high dimensionality of the design space.
This paper introduces Conditioning by Adaptive Sampling (CbAS), a novel framework that addresses these challenges by estimating the distribution of inputs conditioned on target properties. Unlike gradient-based methods requiring differentiability, CbAS employs importance sampling and generative models, such as VAEs, to iteratively refine the input distribution. The core idea is to gradually approximate the conditional distribution of inputs that satisfy the desired property constraints, even when these are rare events. This is achieved by constructing a sequence of relaxed events and importance sampling distributions, which progressively focus the sampling towards the target region.
Experimental results on protein fluorescence data demonstrate that CbAS significantly outperforms existing methods, achieving approximately 15% higher maximum fluorescence and reducing the generation of unrealistic sequences by over 20%. The method's robustness and flexibility make it suitable for a wide range of molecular design tasks, including drug discovery and material science. Its theoretical foundation guarantees convergence and stability, providing a reliable tool for scientific discovery.
Despite its strengths, the approach faces limitations such as computational cost and dependence on the quality of generative models. Future work aims to extend the framework to multi-objective optimization, incorporate reinforcement learning, and broaden its application scope. Overall, CbAS marks a substantial step forward in the integration of probabilistic modeling and optimization for complex scientific problems, promising accelerated innovation in molecular engineering.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备一道特别的菜，你希望这道菜既好吃又符合你的口味。以前，你可能会反复试验不同的调料比例，花费很多时间和精力。而现在，有个聪明的助手可以帮你。它会根据你喜欢的味道，逐步调整调料的用量，每次都比上次更接近理想的味道。它会记住哪些比例效果最好，然后继续优化，直到找到最完美的配比。这个助手不用你告诉它每一步怎么做，只需要告诉它你喜欢的味道，它就会帮你逐步逼近最好的方案。这样，你就能更快做出美味的菜，而且还不用担心试错太多。这个方法就像你有个聪明的厨师助手，帮你一步步找到最棒的调料比例，让做菜变得简单又有趣。
Glossary
Importance Sampling (重要采样)
一种统计技术，用于在稀疏事件中估算目标分布，通过调整采样概率以提高效率。
在算法中用以改善稀疏事件条件化的采样效果。
Variational Autoencoder (变分自编码器)
一种深度生成模型，通过潜在空间编码高维数据，便于生成和条件采样。
作为先验模型，用于估算“真实”样本的分布。
KL散度
衡量两个概率分布差异的指标，优化目标是最小化两者的差异。
在模型逼近目标条件分布时使用。
稀疏事件
在高维空间中发生频率极低的事件，难以通过普通采样估算。
本算法的核心挑战之一。
黑箱预测器
输出结果不可微或无法直接访问内部结构的预测模型。
算法设计中无需依赖模型可微性。
Open Questions Unanswered questions from this research
- 1 如何在极端稀疏事件（如极端高值）条件下保持采样效率？未来是否能结合强化学习进一步提升性能？
- 2 多目标优化和多属性联合条件化的理论基础和实践方法仍需深入探索。
Applications
Immediate Applications
蛋白质工程
利用CbAS快速生成高性能蛋白序列，助力药物开发和酶工程。
药物设计
在药物候选分子筛选中，条件化采样帮助找到具有特定药理性质的分子。
Long-term Vision
材料科学创新
通过条件化生成新型高性能材料，推动可持续能源和电子器件发展。
Abstract
We present a new method for design problems wherein the goal is to maximize or specify the value of one or more properties of interest. For example, in protein design, one may wish to find the protein sequence that maximizes fluorescence. We assume access to one or more, potentially black box, stochastic "oracle" predictive functions, each of which maps from input (e.g., protein sequences) design space to a distribution over a property of interest (e.g. protein fluorescence). At first glance, this problem can be framed as one of optimizing the oracle(s) with respect to the input. However, many state-of-the-art predictive models, such as neural networks, are known to suffer from pathologies, especially for data far from the training distribution. Thus we need to modulate the optimization of the oracle inputs with prior knowledge about what makes `realistic' inputs (e.g., proteins that stably fold). Herein, we propose a new method to solve this problem, Conditioning by Adaptive Sampling, which yields state-of-the-art results on a protein fluorescence problem, as compared to other recently published approaches. Formally, our method achieves its success by using model-based adaptive sampling to estimate the conditional distribution of the input sequences given the desired properties.