ProtBert · AIxSci

What it is

Proteins are long chains built from twenty kinds of building block called amino acids, usually written out as a string of letters. ProtBert treats that string rather like a sentence. It was trained on huge collections of known protein sequences with a simple exercise: hide some of the letters and predict what they were. Doing this over and over forces the model to pick up which arrangements of amino acids tend to occur together. ProtBert belongs to a family of such models, and uses a design first built for ordinary text, called BERT, which reads a sequence in both directions at once rather than only left to right.

Once trained, the model is used as a translator into numbers. Feed it a protein and it returns an embedding: a long list of numbers that stands for the sequence, or for one position in it. Nearby lists tend to mean proteins the model judges similar. Researchers can use these numbers just as they are, as fixed input to a smaller program that predicts something specific, or they can carry on training the model itself on their own task, which is called fine-tuning.

Why scientists use it

Experiments that show what a protein does, or what shape it folds into, are slow and costly, so most known sequences have never been studied directly. Sequences themselves, by contrast, are abundant. A model trained on sequences alone therefore offers a cheap way to bring that abundance to bear: it supplies a general-purpose description of any protein, learned without anyone having to label it first.

That matters because the alternative is hand-made description. Traditionally a researcher would decide which features of a sequence to measure, or would search for close relatives and borrow what is known about them. Embeddings stand in for both. They let a small, ordinary prediction model be trained on a modest set of labelled examples, because the hard part, representing the protein, has already been done. The same embeddings serve very different questions, from finding distant relatives to marking single positions along a chain.

In the record

structural-biologyNeural network assigns protein domains to structural families from sequence aloneEach protein domain sequence was turned into a fixed-length embedding by ProtBERT, used as released, to help detect distant relatives within structural superfamilies.structural-biologyProtein language model trained to spot lactylated lysines in rice proteinsOnly the first four encoder layers were kept and fine-tuned with the rest of the network, giving one vector per window of residues for predicting chemical modification sites.structural-biologyModel labels which protein residues are active sites and what job they doProtBert supplied the sequence side of a model that located and classified the active sites where proteins do their chemical work.

Limits

The model has only ever seen sequences, so it knows nothing of the conditions a protein actually sits in, nor of what it does. Similar numbers mean the sequences look alike by the model's reckoning, which need not mean alike in behaviour. Its view of proteins reflects which ones have been sequenced, so poorly sampled families are served worse. And the numbers are hard to interpret: a prediction built on them can be accurate without telling anyone why, and wrong without warning.

Written by AIxSci: the background from general knowledge, the examples from checked records.