Biology & medicine

Single-Cell Perturbation Prediction

Predict single-cell CRISPRi responses in unseen cellular contexts.

Task & evaluation

Objective
Predict CRISPRi responses in cellular contexts never seen perturbed.
Input
Cellular context and a CRISPRi perturbation.
Output
Predicted single-cell expression responses.
Data
Four public training contexts and three anonymized held-out contexts.
Evaluation
Official cell-eval metrics, normalized and aggregated across held-out contexts.
Research challenge
Separate transferable perturbation effects from context-specific cell behavior.

Understanding the task

Predict how cells respond to a gene perturbation when the new context has never been perturbed in the training data.

What happens when a gene is switched down?

Reducing a gene’s activity can disrupt an entire network of cellular processes. The same intervention may strongly affect one cell type and barely affect another. Can a model predict that response in a new cell context when it has seen only the unperturbed cells?

This zero-shot task tests two forms of transfer: familiar interventions in unfamiliar contexts, and targets whose perturbations were never observed during training. Memorizing an average response can help with the first, but cannot directly solve the second.

Interactive · synthetic example

Transfer the response, not the cell identity

Move the context away from training, then vary the predicted effect. Watch the cell distribution and the same six genes change together.

Model inputs: new basal cells, target identity, chemistry and permitted priors. No perturbed cells from this context.

1 · From basal cells to a predicted populationCyan: basal input · amber: predicted response
Projection axis 1Projection axis 2

Each dot is a synthetic cell. This fixed 2D projection shows distribution shifts; it is neither a biological causal model nor the scoring space.

2 · Inspect the response gene by geneExpression and deltas use illustrative units
Basal → predicted → expression deltaEnable JavaScript to explore six genes.

Δ = predicted mean − basal mean. Direction and magnitude matter: copying basal cells gives Δ = 0, the no-effect baseline.

Evaluator view · synthetic reference only

The reference response is hidden from the model in a new context.

Pearson delta: —Mean |delta error|: —

Pearson delta compares the two gene-response vectors. It can stay high even when magnitude is wrong. The actual benchmark averages seven metrics, each normalized against no effect; these toy diagnostics are not a benchmark score.

1 · Context + intervention

Describe the unperturbed cell population and the protein associated with the target gene.

2 · Trained transition

A transformer combines context and target information to predict a changed cell population.

Protein features let an unseen target enter the model without an observed perturbation.

3 · Count-scale output

Return predicted expression counts. Genes outside the learned panel retain their unperturbed values.

Hidden perturbed cells enter only the evaluator.

Scoring lab · illustrative numbers

From raw metric to seven-part reward

Hold five normalized components fixed at 0.20, 0.10, 0.30, 0.00 and 0.40. Adjust raw Pearson-delta and MAE; their no-effect baselines are fixed at 0.20 and 0.40 respectively.

For the final reward, the near/mid/far final-subset context scores are averaged equally. Here the other two context scores are fixed at 0.10 and 0.05. Scores below the no-effect baseline are clipped to zero, so a zero component means no credited improvement, not necessarily no error.

Learn from four contexts, predict three new ones

Training includes single-cell expression measurements from K562, Jurkat, HCT116 and HEK293T, before and after gene perturbations. Expression counts estimate how many RNA molecules from each gene were detected in a cell. Averaging many cells reveals a population response while individual measurements show variability.

Three anonymized test contexts provide only unperturbed cells and intervention requests. Eighty target genes are withheld from all training perturbations. The most distant context also uses different measurement chemistry, so biological and technical changes are entangled. The interactive cells above are synthetic.

Start with the simplest transferable effect

The mean-effect baseline learns how each known target changes average expression across training contexts, then adds that change to cells from the new context. For an unseen target, it predicts no effect. A gated version suppresses responses when relevant genes are rarely detected.

The learned STATE model instead combines a population of basal cells with a protein-based representation of the target gene. A transformer predicts the changed population. Protein features give an unseen target a meaningful description even without an observed intervention, creating a route to novel-target transfer.

Beat the prediction that nothing changes

A model must improve on copying the unperturbed cells. Seven measures examine complementary properties: response magnitude, response direction, discrimination between perturbations and recovery of genes whose expression changes.

Each measure is converted to skill relative to that no-effect baseline. For example, reducing mean absolute error from 0.40 to 0.30 gives 25% skill. A zero component means no credited improvement over the baseline, not perfect prediction or necessarily no biological effect. Final scores average across the three contexts.

The simple average is hard to beat

The mean-effect model scores about 0.103 overall, compared with 0.046 for the learned STATE model. Its advantage is strongest in the nearer contexts. But it receives zero normalized credit for one measure of changed-gene recovery in every context: a useful aggregate score can coexist with an important failure.

STATE shows a narrower strength on previously unseen targets in the far context, with a reported score of 0.12 versus zero for the mean-effect model. This supports a signal of novel-target transfer, not uniformly accurate expression prediction. Complete context-holdout tests and repeated training runs are not available for STATE.

Skill relative to predicting no effect; higher is better
SystemOverallNearMidFar
No effect0.0000.0000.0000.000
Average response0.1030.1500.1180.042
Average response with activity filter0.1020.1500.1180.039
Learned response (STATE)0.0460.0630.0330.043

Compute and resources

Average-response table

Reported measurement

<2 min
Time
<2 min
Hardware
Not reported
GPU allocation
Not separately reported

Reported preparation time for the simple baseline.

No proportional breakdown: only a range, bound or per-decision timing is reported.

STATE training

Reported measurement

≈1.5 h / run
Time
≈1.5 h / run
Hardware
1 × H100
GPU allocation
Not separately reported

Reported training time for the learned response model.

Elapsed time by stage

STATE training (uncontended)
90 min

No finer stage timing is reported for this workload. Separate cards describe separate measurements, not parts of one total.

About 1.5 hours on an uncontended H100; the source also reports 4.5 hours under load. Data preparation is additional.

Average-response prediction

Reported measurement

8–30 s / context
Time
8–30 s / context
Hardware
Not reported
GPU allocation
Not separately reported

Reported inference time for the simple mean-effect baseline.

No proportional breakdown: only a range, bound or per-decision timing is reported.

Three-context scoring

Reported measurement

≈25 min
Time
≈25 min
Hardware
One-GPU verifier configuration
GPU allocation
Not separately reported

Reported scoring time for three contexts.

Elapsed time by stage

Three-context scoring
25 min

No finer stage timing is reported for this workload. Separate cards describe separate measurements, not parts of one total.

The 60-minute allowance per context is a limit, not the observed scoring time.

Resource details
Prepared training data
≈51 GBFour training contexts; training runs add about 2 GB.
Published model package
≈1.6 GBTraining-only data and intermediate checkpoints are excluded.

Learn when an effect should transfer

An intervention has both a response pattern and a context-dependent strength. Separating these could help the model reuse conserved biology without inventing strong responses in unresponsive cells. Compare a learned gate with the same ungated response before changing the whole architecture.

Other useful controls replace protein features with target identifiers, or compare matched measurement chemistries where possible. These distinguish biological transfer from memorization and technical effects. The present far-context gap alone cannot tell us which explanation dominates.

References and provenance

Predicting cellular responses to perturbation across diverse contexts with State

Introduces the model family and the problem of predicting how cells respond to perturbations in new contexts.

Task definition and experimental evidence

Defines the smaller gene-space reference model used here. It uses protein features, not the available SE embeddings, and its measurements are separate from the published full model.

Task description and code

Code links require access to the project’s GitHub repository.

Leaderboard

Simulated

Design preview · leaderboard values and research histories are simulated.

Agent search and reference methods ranked together by Cell-eval reward.
#MethodTypeCell-eval reward ↑Expression correlation ↑Perturbation recall ↑Research timeRun
1Agent search0.55510.52480.686376 h
2Agent search0.55260.52240.683260 h
3STATE zero-shot (B2)Reference0.55000.52000.6800——
4Cross-context mean delta (B1)Reference0.45100.42640.5576——
How to read these leaderboard metrics

These leaderboard values and research histories are simulated design examples. The leaderboard is ranked by Cell-eval reward. The research curves show relative improvement in this metric over STATE zero-shot (B2); only accepted new-best results update the retained trajectory. A positive gain means a better result, with the sign adjusted for metrics where lower is better. The task protocol is described in Task & evaluation above. These preview columns and trajectories are not measured benchmark results and must not be compared with the recorded experiments. The leaderboard and search traces below are simulated interface examples, not verified experiments. The real protocol averages seven no-effect-normalized cell-eval 0.8.2 metrics over each context’s 200 final requests, then averages three contexts. Individual correlations and other display metrics are not interchangeable with that reward. Verified baseline results and their limits appear in the task guide.

Cell-eval rewardPrimary metric · Higher is better
The task-defined aggregate evaluation for predicted cellular perturbation responses. Component measures capture expression agreement and perturbation-specific behavior.
Expression correlationHigher is better
Correlation between predicted and observed gene-expression responses under the task’s cell or population aggregation.
Perturbation recallHigher is better
Recovery of the reference perturbation effects under the protocol’s detection criteria, measuring coverage rather than only expression correlation.