Transformers in astronomy and astrophysics

How it is used

Astronomy produces long sequences: a star's brightness measured night after night, a spectrum spread across wavelength, a sky survey tiled into millions of small images. A transformer is a network built around attention, a step that lets every element of the input look at every other and decide how much each one matters. There are no fixed neighbourhoods, so a feature at one end of a sequence can bear directly on a feature at the other. In this field that machinery appears in two guises: as the whole model, taking a light curve or an image cut into patches as its input, and as attention layers inserted inside an otherwise conventional image network.

The work it is given is mostly detection, classification and generation. Models are pointed at survey cutouts to find sources and draw boxes around them, at brightness measurements to sort transient events or measure rotation, and at image sets to produce synthetic frames resembling real observations. Because labelled astronomical data are scarce, these models are often trained on simulated data, or trained first on a large generic image collection and then adjusted on a smaller astronomical one.

Why researchers use it

The appeal is that attention does not assume where the relevant information sits. Astronomical signals are irregular: observations are interrupted by daylight, weather and satellite orbits, so measurements arrive at uneven intervals and with uneven quality. A radio source and the galaxy that hosts it may be separated across a tile and visible only in different wavebands. Methods that assume evenly spaced samples, or that only combine nearby pixels, must have those gaps patched over first. A model that weighs every element against every other can, in principle, learn the long-range associations itself.

It also stands in for human inspection. Surveys now report more candidate events each night than anyone can look at, and the older alternative was a hand-built set of rules or measured features. Transformer-based models are used instead as a filter, ranking or labelling candidates so that people and telescopes are spent on the interesting ones.

In the record

astronomyA search engine for radio galaxy shapes, built on a fine-tuned image-text modelRadio tiles and matching infrared mosaics were cut around source positions and passed through an object detection model, producing boxes, categories, confidence scores and candidate infrared hosts.astronomyDiffusion model generates synthetic images of the Sun sorted by flare strengthA diffusion model whose image backbone carried self-attention layers between its downsampling and upsampling stages was trained on solar ultraviolet images to generate synthetic frames.astronomyAstronomers test pre-trained image models for sifting telescope alertsAlert images from a time-domain survey were paired with colour images from a deep imaging survey at matching field of view and pixel scale, for pre-training then fine-tuning.astronomyNeural networks sort supernovae and estimate their distances from brightness aloneSimulated supernova light curves were turned into flux and uncertainty heatmaps by fitting a Gaussian process across wavelength and time, then sampling it on a fixed grid.

Limits

Attention over every pair of elements is expensive, so inputs are usually shrunk, cropped or resampled onto a coarse grid, and fine detail is lost before the model sees anything. These models need far more training examples than astronomy usually labels, which pushes researchers towards simulations or other surveys; a model tuned on either can fail on real data whose noise and instrument quirks differ. Confidence scores are not probabilities, and a plausible synthetic image is not evidence about the Sun. What the attention has latched onto is rarely recoverable, so errors are hard to anticipate.

Written by AIxSci: the background from general knowledge, the examples from checked records.