Protein language models in structural biology and protein science

How it is used

Structural biology and protein science ask what shape a protein folds into, how stable that shape is, which other proteins it sticks to, and which chemical tags get attached to it. A protein is a chain of amino acids, written as a string of letters. A protein language model is trained on vast numbers of such strings, learning which patterns of letters tend to occur together, much as a text model learns which words go together. The result is a numerical description, called an embedding, of each position in a chain and of the chain as a whole. Researchers feed these descriptions into further models that predict a property: a modification site, a stability change, a binding effect.

In the examples, models of this kind are used while frozen, meaning their internal settings are left untouched and only their output descriptions are used. One study reads out both the per-position descriptions and the model's internal attention patterns, which record how strongly the model relates one position to another, and uses them to pick out recurring short stretches of sequence. Another combines window-averaged and whole-chain descriptions from three different models with ordinary chemical properties of the amino acids, then stacks several predictors on top. A third uses a model conditioned on structure to score mutations. Related work in the same field uses structure prediction and graph models instead, over experimental and predicted structures.

Why researchers use it

Experiments are the slow step. Measuring a binding strength, a folding stability or a modification site takes laboratory work for each case, so measured data covers only a small part of the space researchers want to explore. Sequences, by contrast, are abundant: public databases hold far more sequences than solved structures or measured properties. A protein language model turns that abundance into something usable, because it is trained without labels and so can learn from every sequence available. What it stands in for is a hand-built description of a protein. Previously a researcher had to decide which chemical properties, alignments or windows mattered; the model supplies a general-purpose description instead.

That description is also cheap to reuse. Once trained, a model can be run over new sequences and its output stored, so a small predictor can be fitted on whatever measured data exists. Several examples do exactly this, keeping the large model fixed and training only a light layer on top, sometimes alongside the older hand-built features rather than instead of them. The models can also be read as a source of information about the protein itself, since the patterns they have learned reflect constraints that evolution imposed on real sequences.

In the record

structural-biologyAttention patterns inside a protein language model used to cut sequences into reusable unitsESM2-650M was run without fine-tuning on each sequence to obtain attention matrices and per-residue embeddings, which were analysed to define recurring sequence units.structural-biologyLanguage models trained on protein sequences used to predict sugar-attachment sitesPer-residue, window-averaged and whole-sequence embeddings from three frozen models, plus physicochemical parameters, were used as features for stacked predictors of glycosylation sites.structural-biologyAdding a term for the unfolded protein sharpens AI stability predictionsESM-IF1, conditioned on structure, scored each mutation over wild-type and mutant structures, supplying the folded-state term of a stability prediction.structural-biologyDeep learning tool predicts how single mutations weaken protein partnershipsMeasured binding free energy changes were paired with experimental and modelled structures, and augmented with hypothetical reverse mutations, to train a graph-based predictor.

Limits

These models learn from sequences that exist, so they reflect what evolution has produced rather than the physics of folding. They say nothing directly about a structure, and the examples that need one still rely on experimental structures or separate prediction tools. A frozen model gives a general description, not one tuned to the property being predicted, and the predictor sitting on top is still limited by how little measured data exists and by how that data was gathered. Chains are also truncated at a fixed length and restricted to standard amino acids, so unusual residues and long proteins fall outside what the model handles.

Written by AIxSci: the background from general knowledge, the examples from checked records.