Transformers in materials and computational chemistry

How it is used

Materials and computational chemistry try to predict what a substance will do from what it is made of and how its atoms are arranged. A transformer takes an input broken into pieces, such as the characters of a chemical formula, the atoms of a molecule or the sites in a crystal, and learns how strongly each piece should attend to every other piece before producing an answer. In this field the pieces are usually atoms, bonds, repeating units or composition entries, and the answer is a property: how much gas a porous framework will take up, how dense a crystal will be, how hard an alloy is, or which starting chemicals and furnace temperatures a synthesis needs.

The attention step can also be shaped by physics rather than learned from scratch. Instead of letting the model discover which atoms matter to each other, researchers compute distances, angles and dihedral angles between atoms and turn them into energy-like terms that serve as the attention weights. Transformers trained on text are used differently again: a language model is simply asked, in words, for a ranked list of precursor chemicals or for processing temperatures in a structured format, with its answers tidied up by chemistry software and scored against recorded syntheses.

Why researchers use it

The alternative is calculation from first principles or from simulated molecular motion, which is accurate but slow. Screening a large library of candidate frameworks, polymers or alloys by simulation can be impractical, so a trained model stands in for the simulation: it is fitted to structures whose properties are already known, from measurement, from curated databases or from simulations run in advance, and then used to guess at new ones cheaply. Attention suits this because chemical behaviour is relational. What one atom does depends on its neighbours and sometimes on distant parts of the structure, and attention lets every part of an input condition every other part without a fixed notion of locality.

In the record

Limits

These models are only as good as the data behind them, and much of that data is either simulated, so it inherits the simulation's assumptions, or mined from published text, where recipes are recorded unevenly. Chemical space is vast and training sets cluster around what has already been made, so predictions for genuinely unfamiliar compositions are unreliable. Attention weights are often read as explanations, but they show what the model used, not why a material behaves as it does. Splitting data carelessly, so that near-identical structures appear in training and testing, flatters accuracy.

Written by AIxSci: the background from general knowledge, the examples from checked records.

Related

• Transformer: the idea, across every field
• AI in materials and computational chemistry

Every article