Single-Cell Perturbation Prediction
Predict single-cell CRISPRi responses in unseen cellular contexts.
Task & evaluation
- Objective
- Predict CRISPRi responses in cellular contexts never seen perturbed.
- Input
- Cellular context and a CRISPRi perturbation.
- Output
- Predicted single-cell expression responses.
- Data
- Four public training contexts and three anonymized held-out contexts.
- Evaluation
- Official cell-eval metrics, normalized and aggregated across held-out contexts.
- Research challenge
- Separate transferable perturbation effects from context-specific cell behavior.
Understanding the task
Predict how cells respond to a gene perturbation when the new context has never been perturbed in the training data.
What happens when a gene is switched down?
Reducing a gene’s activity can disrupt an entire network of cellular processes. The same intervention may strongly affect one cell type and barely affect another. Can a model predict that response in a new cell context when it has seen only the unperturbed cells?
This zero-shot task tests two forms of transfer: familiar interventions in unfamiliar contexts, and targets whose perturbations were never observed during training. Memorizing an average response can help with the first, but cannot directly solve the second.
Transfer the response, not the cell identity
Move the context away from training, then vary the predicted effect. Watch the cell distribution and the same six genes change together.
Model inputs: new basal cells, target identity, chemistry and permitted priors. No perturbed cells from this context.
Each dot is a synthetic cell. This fixed 2D projection shows distribution shifts; it is neither a biological causal model nor the scoring space.
Δ = predicted mean − basal mean. Direction and magnitude matter: copying basal cells gives Δ = 0, the no-effect baseline.
The reference response is hidden from the model in a new context.
Pearson delta compares the two gene-response vectors. It can stay high even when magnitude is wrong. The actual benchmark averages seven metrics, each normalized against no effect; these toy diagnostics are not a benchmark score.
Describe the unperturbed cell population and the protein associated with the target gene.
A transformer combines context and target information to predict a changed cell population.
Protein features let an unseen target enter the model without an observed perturbation.
Return predicted expression counts. Genes outside the learned panel retain their unperturbed values.
Hidden perturbed cells enter only the evaluator.
From raw metric to seven-part reward
Hold five normalized components fixed at 0.20, 0.10, 0.30, 0.00 and 0.40. Adjust raw Pearson-delta and MAE; their no-effect baselines are fixed at 0.20 and 0.40 respectively.
For the final reward, the near/mid/far final-subset context scores are averaged equally. Here the other two context scores are fixed at 0.10 and 0.05. Scores below the no-effect baseline are clipped to zero, so a zero component means no credited improvement, not necessarily no error.
Learn from four contexts, predict three new ones
Training includes single-cell expression measurements from K562, Jurkat, HCT116 and HEK293T, before and after gene perturbations. Expression counts estimate how many RNA molecules from each gene were detected in a cell. Averaging many cells reveals a population response while individual measurements show variability.
Three anonymized test contexts provide only unperturbed cells and intervention requests. Eighty target genes are withheld from all training perturbations. The most distant context also uses different measurement chemistry, so biological and technical changes are entangled. The interactive cells above are synthetic.
Start with the simplest transferable effect
The mean-effect baseline learns how each known target changes average expression across training contexts, then adds that change to cells from the new context. For an unseen target, it predicts no effect. A gated version suppresses responses when relevant genes are rarely detected.
The learned STATE model instead combines a population of basal cells with a protein-based representation of the target gene. A transformer predicts the changed population. Protein features give an unseen target a meaningful description even without an observed intervention, creating a route to novel-target transfer.
Beat the prediction that nothing changes
A model must improve on copying the unperturbed cells. Seven measures examine complementary properties: response magnitude, response direction, discrimination between perturbations and recovery of genes whose expression changes.
Each measure is converted to skill relative to that no-effect baseline. For example, reducing mean absolute error from 0.40 to 0.30 gives 25% skill. A zero component means no credited improvement over the baseline, not perfect prediction or necessarily no biological effect. Final scores average across the three contexts.
The simple average is hard to beat
The mean-effect model scores about 0.103 overall, compared with 0.046 for the learned STATE model. Its advantage is strongest in the nearer contexts. But it receives zero normalized credit for one measure of changed-gene recovery in every context: a useful aggregate score can coexist with an important failure.
STATE shows a narrower strength on previously unseen targets in the far context, with a reported score of 0.12 versus zero for the mean-effect model. This supports a signal of novel-target transfer, not uniformly accurate expression prediction. Complete context-holdout tests and repeated training runs are not available for STATE.
| System | Overall | Near | Mid | Far |
|---|---|---|---|---|
| No effect | 0.000 | 0.000 | 0.000 | 0.000 |
| Average response | 0.103 | 0.150 | 0.118 | 0.042 |
| Average response with activity filter | 0.102 | 0.150 | 0.118 | 0.039 |
| Learned response (STATE) | 0.046 | 0.063 | 0.033 | 0.043 |
Compute and resources
Average-response table
Reported measurement
- Time
- <2 min
- Hardware
- Not reported
- GPU allocation
- Not separately reported
Reported preparation time for the simple baseline.
STATE training
Reported measurement
- Time
- ≈1.5 h / run
- Hardware
- 1 × H100
- GPU allocation
- Not separately reported
Reported training time for the learned response model.
Elapsed time by stage
- STATE training (uncontended)
- 90 min
No finer stage timing is reported for this workload. Separate cards describe separate measurements, not parts of one total.
About 1.5 hours on an uncontended H100; the source also reports 4.5 hours under load. Data preparation is additional.
Average-response prediction
Reported measurement
- Time
- 8–30 s / context
- Hardware
- Not reported
- GPU allocation
- Not separately reported
Reported inference time for the simple mean-effect baseline.
Three-context scoring
Reported measurement
- Time
- ≈25 min
- Hardware
- One-GPU verifier configuration
- GPU allocation
- Not separately reported
Reported scoring time for three contexts.
Elapsed time by stage
- Three-context scoring
- 25 min
No finer stage timing is reported for this workload. Separate cards describe separate measurements, not parts of one total.
The 60-minute allowance per context is a limit, not the observed scoring time.
Resource details
- Prepared training data
- ≈51 GBFour training contexts; training runs add about 2 GB.
- Published model package
- ≈1.6 GBTraining-only data and intermediate checkpoints are excluded.
Learn when an effect should transfer
An intervention has both a response pattern and a context-dependent strength. Separating these could help the model reuse conserved biology without inventing strong responses in unresponsive cells. Compare a learned gate with the same ungated response before changing the whole architecture.
Other useful controls replace protein features with target identifiers, or compare matched measurement chemistries where possible. These distinguish biological transfer from memorization and technical effects. The present far-context gap alone cannot tell us which explanation dominates.
References and provenance
Predicting cellular responses to perturbation across diverse contexts with State
Introduces the model family and the problem of predicting how cells respond to perturbations in new contexts.
Task definition and experimental evidence
Defines the smaller gene-space reference model used here. It uses protein features, not the available SE embeddings, and its measurements are separate from the published full model.
Code links require access to the project’s GitHub repository.
Leaderboard
SimulatedDesign preview · leaderboard values and research histories are simulated.
| # | Method | Type | Cell-eval reward ↑ | Expression correlation ↑ | Perturbation recall ↑ | Research time | Run |
|---|---|---|---|---|---|---|---|
| 1 | Agent search | 0.5551 | 0.5248 | 0.6863 | 76 h | ||
| 2 | Agent search | 0.5526 | 0.5224 | 0.6832 | 60 h | ||
| 3 | STATE zero-shot (B2) | Reference | 0.5500 | 0.5200 | 0.6800 | — | — |
| 4 | Cross-context mean delta (B1) | Reference | 0.4510 | 0.4264 | 0.5576 | — | — |
How to read these leaderboard metrics
These leaderboard values and research histories are simulated design examples. The leaderboard is ranked by Cell-eval reward. The research curves show relative improvement in this metric over STATE zero-shot (B2); only accepted new-best results update the retained trajectory. A positive gain means a better result, with the sign adjusted for metrics where lower is better. The task protocol is described in Task & evaluation above. These preview columns and trajectories are not measured benchmark results and must not be compared with the recorded experiments. The leaderboard and search traces below are simulated interface examples, not verified experiments. The real protocol averages seven no-effect-normalized cell-eval 0.8.2 metrics over each context’s 200 final requests, then averages three contexts. Individual correlations and other display metrics are not interchangeable with that reward. Verified baseline results and their limits appear in the task guide.
- Cell-eval rewardPrimary metric · Higher is better
- The task-defined aggregate evaluation for predicted cellular perturbation responses. Component measures capture expression agreement and perturbation-specific behavior.
- Expression correlationHigher is better
- Correlation between predicted and observed gene-expression responses under the task’s cell or population aggregation.
- Perturbation recallHigher is better
- Recovery of the reference perturbation effects under the protocol’s detection criteria, measuring coverage rather than only expression correlation.
AI4AI research progress
Research trajectory
Runtime–performance frontier
How improvement is measured
Relative improvement over STATE zero-shot (B2) in the primary metric; only retained results update the best-so-far curve.
Accepted new-best experiments raise the trajectory. The frontier includes evaluated candidates; trial, partial and stopped experiments are excluded.