Interpretation

What it is

A model trained on scientific data is, at bottom, a function: data goes in, an answer comes out. Interpretation treats that function as an object of study in its own right. Researchers ask which parts of the input the model leaned on, what it has grouped together, what internal quantities it has come to represent, and what its mistakes have in common. The methods vary. Some hide or remove part of the input and watch how the answer changes, which shows what the model depended on. Some reduce many measurements to a few underlying directions of variation, so a pattern becomes visible on a page. Some inspect the model's internal numbers directly, or fit a simple, readable model to the complicated one's behaviour.

The result of interpretation is usually a statement in the language of the science, not of the software. A model that sorts images into categories can be asked which categories cluster together, and the clusters may correspond to physical causes. A model trained to predict a property from a structure can be asked what it treats as similar, and the answer may amount to a rule a chemist could state. Interpretation also covers reading an automated search: looking back over what an experimental system chose to try, and inferring the shape of the landscape it was moving across.

Why scientists use it

Prediction alone often does not settle a scientific question. Scientists want a mechanism, a rule, or a reason to trust the answer, and a model that is accurate without being legible gives none of those. Interpretation is the step that converts accuracy into understanding. It also does practical work: it provides evidence that a model is right for the right reasons rather than exploiting some accident of how the data were collected, which matters when the model's output will be used to make a claim.

Interpretation stands in for an experiment that would be slow, expensive or impossible. If a model has absorbed the regularities of a very large body of measurements, examining the model can be a cheap way of asking what those regularities are, where measuring each case directly would take years. In this sense the trained model serves as a compressed summary of the data, and interpretation is the act of reading that summary back out.

In the record

Limits

Interpretation explains the model, not necessarily nature. A model can be accurate for the wrong reasons, and a tidy explanation of its behaviour may describe a quirk of the training data rather than a real mechanism. Methods often disagree with each other, and the same model can yield different stories depending on which technique is used. What emerges is a correlation the model found useful, which is not the same as a cause. Any interpretation remains a hypothesis until an independent experiment tests it.

Written by AIxSci: the background from general knowledge, the examples from checked records.

Every article