PerturbCellRL: Verifier-Guided Reinforcement Learning for Single-Cell Perturbation Prediction
PerturbCellRL uses four biological verifiers to post-train scDFM, improving single-cell consistency after 1,600 RL steps while preserving population-level competitiveness.
Key Findings
Methodology
PerturbCellRL treats pretrained scDFM as a policy and applies online reinforcement learning through DiffusionNFT. For each control cell and perturbation, it samples multiple candidates, scores them with Pearson top-k, RMSE top-k, DE Spearman, and Pathway activity verifiers, then updates the flow velocity toward high-reward trajectories. A KL penalty keeps the post-trained policy close to scDFM.
Key Results
- On Norman additive and holdout protocols, PerturbCellRL improves all four optimized single-cell rewards over pretrained scDFM during 1,600 training steps and also raises the held-out single-cell Discrimination Score. The supplied text does not reproduce Figure 5's exact numerical values, so no percentage improvement should be invented.
- Across Norman holdout single, holdout double, Norman additive, and ComboSciPlex, the method remains competitive with scDFM and other baselines on population metrics. Pearson Δ, DS, MMD, and Energy Distance were not optimized, providing checks against reward hacking.
- At inference, PROGENy Pathway activity supports best-of-N selection: the highest-scoring candidate is retained. Figure 6 reports gains at both single-cell and population levels, but the supplied text omits exact N values and effect sizes.
Significance
The work shifts the target from generating expression distributions that look realistic to generating individual cells that pass explicit biological checks. This addresses a persistent weakness of unpaired single-cell perturbation data: population agreement can conceal cells with incorrect perturbation directions or implausible mechanisms. For drug screening, target discovery, and combinatorial design, cell-level verification could reduce wet-lab candidates while retaining distributional fidelity.
Technical Contribution
The central technical contribution is to convert nondifferentiable, externally computed biological checks into reinforcement-learning rewards for a conditional flow model. The objective maximizes the weighted reward while constraining D_KL(πθ′||πθ)≤ε. DiffusionNFT operates on the flow forward process and constructs positive and negative velocity targets, avoiding intractable exact likelihoods. The same verifier framework supports training, evaluation, and inference-time selection.
Novelty
The novelty lies less in a new backbone than in a verifier-guided generative-alignment paradigm. Relative to scDFM, CPA, GEARS, STATE, and CellFlow, which primarily target population prediction, PerturbCellRL explicitly optimizes directionality, local manifold proximity, differential-expression ranking, and pathway response. It also reuses a reference-free pathway verifier for test-time scaling.
Limitations
- Pathway activity depends on single-gene annotations and an approximately 680K-parameter PROGENy-predictor MLP. Combinatorial perturbations are currently excluded because reliable pathway directions are difficult to annotate.
- The supplied manuscript text omits the complete numerical tables and figures, preventing rigorous reporting of absolute scores, percentage gains, or best-of-N scaling curves.
- Verifiers are incomplete and may be exploitable. KL regularization preserves the scDFM manifold, but it cannot guarantee causal correctness or eliminate reward hacking.
Future Work
Future work should extend pathway annotations to combinatorial interventions, add cell-type classifiers and housekeeping-gene stability checks, calibrate reward uncertainty, and improve cross-dataset transfer. More efficient best-of-N procedures and prospective wet-lab validation are especially important. The broader direction is to combine learned, curated, and causal verifiers into auditable virtual-cell systems.
AI Executive Summary
Single-cell perturbation models aim to predict how genes or chemicals reshape transcription before expensive experiments. Methods such as scDFM, CPA, GEARS, STATE, and CellFlow can reproduce population-level distributions, but a generated individual cell may still express a biologically wrong perturbation response. The difficulty is fundamental: control and treated cells are usually unpaired, so the model learns a conditional distribution rather than a known before-and-after trajectory.
PerturbCellRL addresses this gap by treating biological checks as rewards. It starts from the pretrained flow-matching generator scDFM and evaluates each candidate with Pearson top-k similarity, RMSE top-k proximity, DE Spearman ranking, and PROGENy-based Pathway activity. DiffusionNFT then increases the probability of high-reward trajectories while a KL penalty prevents excessive drift from the pretrained expression manifold. At inference, multiple candidates can be generated and the one with the best pathway score selected.
On Norman genetic perturbations and ComboSciPlex chemical perturbations, 1,600 RL steps improved the four optimized single-cell rewards and the held-out Discrimination Score. Population-level performance remained competitive across Norman additive, Norman holdout single, Norman holdout double, and ComboSciPlex; Pearson Δ, DS, MMD, and Energy Distance were not training rewards. The supplied text does not provide exact table values. The study therefore offers a compelling alignment framework, while leaving combination annotations, reward hacking, computational cost, and prospective experimental validation unresolved.
Deep Analysis
Background
Single-cell RNA sequencing enabled virtual-cell and perturbation modeling. scVI established probabilistic representation learning, while CPA, GEARS, STATE, and CellFlow explored conditional autoencoders, graph models, transformers, and generative approaches. scDFM represents the flow-matching line. Norman, ComboSciPlex, and the Virtual Cell Challenge provide genetic and chemical benchmarks, yet most methods still emphasize pseudobulk means, distribution distances, or population statistics.
Core Problem
Given normalized control expression u_i and perturbation c, the task is to generate y_i∼πθ(·|u_i,c). Because control and treated cells are generally unpaired, no direct individual response is observed. A model may match the target population while misranking DE genes, drifting in stable genes, or reversing pathway direction in particular cells. These biological objectives are often nondifferentiable, making ordinary backpropagation inadequate.
Innovation
The study contributes four elements. First, it defines cell-level verifiers for similarity, local target-manifold proximity, DE ranking, and pathway direction. Second, it applies DiffusionNFT online RL to a conditional flow model without exact likelihood evaluation. Third, it uses KL regularization to limit reward-induced distributional damage. Fourth, it introduces reference-free PROGENy best-of-N selection, enabling inference-time improvement even without observed treated cells.
Methodology
- �� scDFM learns a velocity field vθ mapping x0∼N(0,I) to perturbed expression; its flow-matching target is ||vθ(x_t,t,u,c)−(y_obs−x0)||², with x_t=(1−t)x0+ty_obs.
- �� Pearson top-k averages correlations to the most similar real target cells; RMSE top-k normalizes distances to nearby target cells; DE Spearman correlates ranks of generated and real fold changes on significant DE genes.
- �� PROGENy contains 14 signaling signatures. An approximately 680K-parameter MLP estimates pathway scores, and Δs=fϕ(y)−fϕ(u) is converted to a signed, confidence-weighted reward σ(wdΔs/τ).
- �� Each group uses 32 rollouts; normalized rewards define positive and negative velocity targets. The objective combines reward-weighted velocity losses with βKL, using equal reward weights by default and learning rate 2×10^-6.
Experiments
Experiments use Norman and ComboSciPlex. Norman includes additive and holdout protocols with four random folds. Baselines are Control, Additive, GEARS, CPA, STATE, CellFlow, and scDFM. Metrics include MAE, Pearson Δ, Pearson Δ̂, DE-Spearman LFC Sig, DS, MMD, Energy Distance, and the four single-cell verifier rewards. Batches contain 64 samples and 32 rollouts per group; KL weights are 2.0 for Norman and 1.2 for ComboSciPlex. Each run lasts 1,600 steps on one H100 GPU.
Results
PerturbCellRL raises all four optimized rewards and the held-out DS on Norman additive and holdout settings, suggesting that gains are not merely reward overfitting. It remains competitive with scDFM at the population level, while Pearson Δ, DS, MMD, and Energy Distance provide unoptimized checks. Best-of-N pathway selection further improves single-cell and population pathway scores. Exact absolute values are absent from the supplied text and should not be fabricated.
Applications
The framework can prioritize compounds by predicted pathway responses, rank genetic targets by DE and pathway direction, and explore candidate responses for perturbation design. It is most useful when a reliable control population, perturbation label, normalization pipeline, and applicable pathway annotation are available. Its value is greatest when wet-lab screening is expensive and candidate space is large.
Limitations & Outlook
Performance depends on verifier validity: PROGENy coverage and annotation confidence may be incomplete, especially for combinations. Best-of-N increases inference cost, while RL may exploit imperfect rewards despite KL regularization. The paper also lacks complete numerical tables in the supplied text and does not establish prospective experimental success. Future work should add causal and uncertainty-aware verifiers, cross-platform tests, combinatorial annotations, and wet-lab validation.
Plain Language Accessible to non-experts
Imagine a central kitchen training a chef to recreate dishes from a famous restaurant. scDFM is already a strong chef: large meals usually resemble the restaurant’s menu, but an individual plate may contain a seasoning that points in the wrong direction. Traditional evaluation tastes the whole buffet on average, so one bad plate can remain hidden.
PerturbCellRL adds four inspectors. One checks whether the main flavor changes in the right direction. Another asks whether the dish is close to real examples from the same menu. A third checks whether the most important ingredients are ranked correctly. The fourth consults a recipe book to determine whether a known intervention should strengthen or weaken a particular flavor. Their scores become feedback for the chef.
The chef makes many versions for the same order. High-scoring versions become more likely in future, while a rule prevents the chef from abandoning everything already learned. Before serving, the kitchen can also prepare N plates and keep the one with the most biologically sensible pathway flavor. This preserves variety rather than turning every answer into one average dish.
Tests on Norman and ComboSciPlex show better individual-cell consistency after 1,600 training steps, while overall meal-level quality remains competitive. Yet inspectors can be incomplete, especially when several ingredients change together. Real laboratory tasting is still essential.
ELI14 Explained like you're 14
Think of a cell simulator as a game that predicts what happens after you switch off a gene or add a drug. You give it an ordinary cell and an instruction, and it creates a possible “after” version. But the game usually sees groups of ordinary cells and treated cells, not the exact same cell before and after. So it can make a cell that looks realistic while reacting in the wrong direction.
PerturbCellRL adds four referees. One checks the overall direction of change, one checks whether the answer is close to real treated cells, one checks whether the biggest gene changes are ranked correctly, and one checks whether known biological pathways should go up or down. The model generates many answers, receives scores, and learns to make high-scoring answers more often without forgetting its original skills.
It can also act like doing several versions of a homework problem and submitting the answer with the best reasoning check. On Norman genetic data and ComboSciPlex chemical data, the four main scores improved after 1,600 training steps. A separate score called DS improved too, even though it was not directly used for training—that is a useful fairness check.
But this is not a magic cheat code! If a referee has incomplete knowledge, the model might learn to impress the referee rather than be truly correct. Predicting several genes changing at once is especially difficult. Scientists still need laboratory experiments to confirm whether the predictions work.
Glossary
Flow matching
A generative method that learns a continuous route from simple noise to a target expression profile. Technically, it fits a time-dependent velocity field rather than directly predicting the final sample.
scDFM supplies the pretrained flow-matching generator used by PerturbCellRL.
Biological verifier
A scoring function that checks whether a generated cell satisfies a specified biological criterion. It may compare data directly or encode curated pathway knowledge.
Four verifiers provide both evaluation metrics and RL rewards.
DiffusionNFT
An online RL method for diffusion and flow models. It uses the forward process to construct contrastive positive and negative velocity targets without requiring tractable exact sample likelihoods.
PerturbCellRL uses DiffusionNFT to update the scDFM velocity field.
PROGENy
A pathway-activity framework based on weighted gene signatures from perturbation experiments. The paper uses 14 curated signaling pathways.
PROGENy supports the Pathway activity reward and reference-free candidate selection.
Discrimination Score
A metric that ranks a predicted perturbation effect against other test perturbations using L1 distance to the real effect. It is independent of the four RL rewards.
DS serves as a held-out check against reward overfitting.
Open Questions Unanswered questions from this research
- 1 Reliable pathway directions for combinatorial perturbations remain unavailable. Progress requires causal networks, literature evidence, and large combination screens that can calibrate signed pathway annotations.
- 2 It is unknown whether higher verifier scores consistently translate into higher prospective experimental success. Cross-platform, cross-cell-type, and wet-lab validation are needed.
Applications
Immediate Applications
Drug-candidate prioritization
Drug teams can generate responses for candidate compounds, score pathway directionality, and prioritize a smaller set for experiments. This requires compatible control data, perturbation labels, normalization, and trustworthy pathway annotations; the expected benefit is reduced screening burden.
Genetic perturbation planning
Researchers can rank target genes by predicted expression effects, DE ordering, and pathway consistency before running CRISPR or overexpression screens. Predictions can guide selection of informative interventions, but independent biological assays remain necessary.
Long-term Vision
Auditable virtual cells
A future virtual-cell platform could combine learned, curated, causal, and cell-identity verifiers. Such a system would generate diverse responses while reporting why each individual prediction is considered biologically plausible, enabling more transparent translational use.
Abstract
Single-cell perturbation models can reduce costly wet-lab screening by predicting how cells respond transcriptionally to interventions. While recent generative models improve population-level prediction, individual generated cells are not explicitly checked for biological consistency. We introduce PerturbCellRL, a reinforcement learning (RL) framework that post-trains a pretrained single-cell transcriptomic generator using a suite of cell-level verifiers as rewards. These verifiers define four rewards: Pearson top-k similarity, RMSE top-k proximity, DE Spearman, and Pathway activity. The Pathway activity verifier rewards cells whose pathway responses match known perturbation biology. We evaluate PerturbCellRL on multiple genetic and chemical perturbation benchmarks. Across these benchmarks, PerturbCellRL improves over the pretrained flow-matching generator on reward-aligned evaluation metrics and a held-out evaluation metric. Moreover, PerturbCellRL remains competitive with state-of-the-art methods on population-level metrics. Together, these results frame trustworthy single-cell prediction as verifier-guided generative alignment, moving beyond matching expression distributions toward predictions whose single-cell perturbation effects are explicitly checked for biological consistency.