Transformers in structural biology and protein science

How it is used

Structural biology and protein science ask what shape a protein takes and what that shape lets it do. A protein is a chain of amino acids, and the chain folds into a compact three-dimensional form. Transformers treat that chain as a sequence, like words in a sentence, and learn which positions matter to which. Trained on very large collections of known protein sequences, they produce a numerical summary, called an embedding, for each position in a chain. Some systems go further and predict the folded coordinates directly. In the examples here, predicted structures were used to decide which loose ends of a protein to trim before an experiment, and to supply measures of how buried each residue is.

The same models are also used without any further training. In one example a protein language model was run on each sequence as it came, and its internal attention patterns, the weights saying which positions it was looking at, were read out alongside per-residue embeddings. In another, embeddings from three such frozen models were combined with simple chemical properties to judge whether a given site on a chain carries a sugar group. The pattern across these examples is that the model is used as a source of features, and conventional statistics or smaller learners make the final call.

Why researchers use it

Measuring a protein structure in the laboratory is slow and does not work for every protein. Sequences, by contrast, are abundant: databases hold vast numbers of them with no structure attached. Transformers are attractive here because they learn from sequences alone, and because the way they relate distant positions in a chain mirrors something real, since residues far apart along the chain often sit next to each other once folded. A model trained this way carries general information about what protein chains look like, and that information can be borrowed for a narrow question with only a modest amount of labelled data.

In the examples, the model stands in for work that would otherwise be done by hand or by experiment. Predicted structures replaced measured ones when deciding where to cut a sequence, and provided contact counts and secondary-structure labels for later analysis. Frozen embeddings replaced hand-designed descriptions of a site's chemical neighbourhood. Attention patterns were used in place of an expert's judgement about which stretches of a chain belong together.

In the record

Limits

These models learn from the sequences people have already collected, so families that are poorly represented are predicted less reliably, and a confident-looking answer carries no guarantee. A predicted structure is a single static picture, which suits a rigid protein better than a floppy one, and it says nothing about the other shapes a chain visits. Embeddings and attention weights are hard to interpret: the numbers work without explaining why. Labelled measurements remain scarce, so a model tuned on them can learn the quirks of one dataset rather than the underlying chemistry.

Written by AIxSci: the background from general knowledge, the examples from checked records.

Related

• Transformer: the idea, across every field
• AI in structural biology and protein science

Every article