Preparation

What it is

Raw measurements are rarely ready for use. Instruments drift, detectors pick up noise, some readings fail outright, and the same quantity may be recorded in different units or on different scales. Preparation is everything done to the data between collecting it and analysing it. Cleaning removes or repairs bad records. Filtering keeps only the measurements that pass a quality test. Normalising rescales values so that different samples, runs or instruments can be compared. Labelling attaches the answer a model is meant to learn, such as what kind of object a signal came from.

Preparation also means deciding what the data will look like to whatever uses it next. Scientists often reduce a long record to a handful of summary quantities, called features, chosen because they carry the information that matters. They may split the data into one part for training a model and a separate part for testing it. Some of these steps are done by hand, some by a rule written in advance, and some by a model trained for the job. Often it is a mixture, with a person checking what the automatic step produced.

Why scientists use it

Analysis assumes that the numbers mean what they claim to mean. A model learns whatever pattern is present in the data it is given, including patterns that come from the instrument rather than from nature. If half the readings are too faint to trust, or if one batch was measured on a warmer day, a model will happily learn that instead of the thing being studied. Preparation is where those problems are removed, or at least made visible, before they turn into a confident wrong answer.

It also stands in for something scarce: a trustworthy set of examples. Many methods need data with the right answer already attached, and that answer usually has to be produced, by a calculation, by an established method or by an expert reading the result. Assembling and checking such a set is often the largest part of the work, and the quality of everything downstream depends on it.

In the record

Limits

Preparation cannot add information that was never measured. Every choice about what to discard is also a choice about what the analysis can see, and a filter set too tightly can remove the rare cases that were the point of looking. Normalising can hide a real difference between batches as easily as it hides an artefact of the instrument. Labels inherited from an older method carry that method's mistakes forward silently. And because these steps are made before the main analysis, their effects are easy to forget and hard to see in the result.

Written by AIxSci: the background from general knowledge, the examples from checked records.

Every article