ProtT5 · AIxSci

What it is

Proteins are chains built from twenty kinds of amino acid, written out as strings of letters. A protein language model is trained on millions of these strings in the way a text model is trained on sentences: parts of the sequence are hidden, and the model learns to guess what belongs there. To do that well it has to pick up regularities in how amino acids follow one another. ProtT5 is one such model, from a family called ProtTrans. After training, it is used not to generate proteins but to describe them. Feed in a sequence and it returns a vector, a long list of numbers, for each amino acid, and these can be averaged to describe a whole protein.

Those numbers are called an embedding. Nothing in them is read directly; they are input for a second, usually much smaller, model trained on a specific question, such as how tightly two proteins stick together. ProtT5 is often left frozen, meaning its own numbers are not changed during that training. Different versions exist, distinguished by the sequence collections they were trained on, and researchers often compute embeddings in several forms: one per amino acid, averaged over a short window around a site of interest, or averaged across the whole chain.

Why scientists use it

The alternative is to hand-pick measurements: the charge of each amino acid, how it behaves in water, how conserved the position is across species. That works, but it depends on knowing in advance what matters. An embedding instead carries whatever the model found useful while learning to predict sequences, which tends to include signals about shape and function that no one wrote down. It also lets small studies borrow from large ones. Measured data about proteins is scarce and costly, while raw sequences are abundant, so the expensive learning happens once, on sequences alone, and many later models reuse it.

Embeddings also make proteins comparable as numbers. Two sequences that share little in their letters can still sit close together in this numerical space if they behave alike, which matters for proteins with no well-studied relatives. And because the output is just a vector, it mixes easily with other kinds of data, such as a chemical structure written as a string, inside one model.

In the record

structural-biologyDeep learning tool predicts how single mutations weaken protein partnershipsSupplied sequence representations for a model predicting how mutations change the binding strength between two proteins, trained on measured mutation records.structural-biologyModel predicts how fast an enzyme works on a substrate, with uncertainty attachedRepresented the enzyme half of each enzyme-and-chemical pair, alongside the substrate written as a SMILES string, to predict catalytic efficiency.structural-biologyLanguage models trained on protein sequences used to predict sugar-attachment sitesOne of three frozen protein language models providing per-residue, window-averaged and whole-sequence embeddings for predicting sites where sugars attach.structural-biologyModel labels which protein residues are active sites and what job they doTwo versions, trained on different sequence collections, used to represent proteins for locating and classifying their active sites.

Limits

An embedding is only as good as what the training sequences contained; proteins unlike anything in that collection are described poorly, and the numbers carry whatever biases the collection had. The vectors are not interpretable: no single number means charge or fold, so a prediction built on them is hard to explain or to check against known chemistry. They describe a sequence, not a structure or an environment, so effects of pH, binding partners or temperature are invisible. And a frozen model cannot learn from the task it is serving.

Written by AIxSci: the background from general knowledge, the examples from checked records.