ESM-2 · AIxSci

What it is

Proteins are long chains built from a small alphabet of building blocks called amino acids. A protein language model is trained the way a text model is trained: it reads vast numbers of real protein chains with parts hidden, and learns to guess what belongs in the gaps. To do that well it has to pick up the regularities of proteins, which blocks tend to sit near which, and which stretches look like a known working part. ESM-2 is one such model.

Once trained, it is usually used as a translator rather than a predictor. A sequence goes in, and out comes an embedding: a long list of numbers for the whole protein, or one list for each position in the chain. Those numbers are then fed to a smaller model trained for a particular question. Most researchers leave ESM-2 unchanged, or frozen, and train only the small model on top.

Why scientists use it

Raw sequences are awkward for machine learning. Counting amino acids throws away order, and treating each position as a separate symbol tells a model nothing about how proteins behave. Embeddings from a protein language model carry some of what the model learned from the whole of nature, so a study with only a modest set of labelled examples can still get somewhere. The expensive training was done once, by someone else, on unlabelled sequences.

This also makes ESM-2 a common baseline and a common ingredient. It stands in for structure and for evolutionary comparison, which both take far more computing time, and its numbers can be combined with older, hand-designed descriptions of a protein's chemistry to see whether the learned features add anything.

In the record

structural-biologyAttention patterns inside a protein language model used to cut sequences into reusable unitsRun unchanged on each sequence to read out its attention matrices and per-residue embeddings, which were then analysed to define recurring protein 'words'.structural-biologyDeep learning trained on robotic peptide mapping predicts where antibody proteins degradeRun over whole antibody sequences to give one vector per asparagine or glutamine site, feeding a small network that predicts chemical damage at that site.structural-biologyProtein language model embeddings tested against classical descriptors for spotting resistance genesUsed with fixed weights to make one averaged vector per protein, compared with, and joined to, classical amino acid and chemical descriptors for resistance prediction.structural-biologyLanguage models trained on protein sequences used to predict sugar-attachment sitesOne of three frozen protein language models supplying per-residue, window-averaged and whole-protein embeddings as features for predicting sugar attachment sites.

Limits

The numbers are not an explanation. An embedding says a position looks unusual, not why, and the model has no notion of a cell, a drug or a laboratory condition. Because it learned from sequences already in public databases, it knows well-studied protein families better than rare ones, and predictions for the latter can look confident while being poor. Embeddings are also not a substitute for measurement: the small model trained on top still needs reliable labels, and it will inherit any bias in how those labels were collected.

Written by AIxSci: the background from general knowledge, the examples from checked records.