Protein Sequence Modeling

A protein’s amino-acid sequence influences its shape, stability, location and interactions. Yet a model that recognizes a fluorescent protein may still struggle to identify its fold or predict a binding partner. These properties demand different kinds of understanding.

PEER tests one shared sequence model across eleven tasks, from enzyme activity and solubility to cellular location and protein interactions. The central question is whether learning these tasks together creates useful common knowledge, or whether their objectives compete.

Some tasks take one sequence and ask for a number or a class. Interaction tasks take two sequences. Secondary structure asks for a label at every amino acid, identifying local structural patterns rather than one property of the whole chain.

The task supplies about 219,000 labeled training examples and a separate unlabeled sequence collection for pretraining. Learned weights start from scratch. Evaluation proteins are kept out of the pretraining pool using sequence-similarity filtering, and hidden test data cannot be used for fitting. Some tests deliberately probe distant folds, making them harder than ordinary random splits.

The baseline first learns to recover hidden amino acids from surrounding sequence. This teaches recurring sequence patterns without requiring a measured biological property. It then learns the eleven supervised tasks together.

For whole-protein properties, residue representations are averaged into a single summary. For interactions, two summaries are combined. Secondary structure instead uses each residue’s representation. Averaging is economical, but it can erase the detailed relationships between residues that form a binding interface.

Large datasets could overwhelm the small ones if every example were sampled equally. The starter uses a compromise that favors larger tasks without giving them proportionate control. Classification and numerical tasks also use different training losses.

A lower sequence-reconstruction loss is not itself proof of better biology. To establish the value of pretraining, compare against supervised training alone at a stated compute budget. To establish the value of sharing, compare task-level performance with appropriate single-task controls.

Classification tasks reward correct labels. Fluorescence, stability and enzyme activity reward the ordering of measured values: can the model identify which protein performs better? Interaction affinity instead penalizes numerical error relative to the training data’s variation.

Adding the same offset to every prediction preserves their order but changes their numerical accuracy. The interactive example demonstrates why one metric cannot answer both questions. All eleven task rewards receive equal weight in the final mean, regardless of dataset size.

Two runs of the small model achieve average rewards of about 0.670 and 0.677, compared with 0.287 for a fixed-statistic control. That control uses dataset-level information rather than making uniformly random guesses.

Binary localization reaches around 90% accuracy, while fold classification remains near 21%. The fold task requires transfer across unfamiliar structural families, so this weakness points to a different challenge from fitting an abundant scalar assay. Two runs show a similar overall pattern, but do not isolate the benefits of pretraining or multitask learning.

Residue-level interactions could help pair tasks; more structured pooling could help fold classification; longer context could help long chains. Compare these changes with matched training exposure and report all tasks, so improvement on an easy target cannot hide damage to a difficult one.