materials-chemistry/ai produced the result/arXiv 2026 · v2
A test suite for judging how well AI reads crystal symmetry from X-ray patterns
Researchers built fixed benchmarks and scoring rules for machine learning that infers crystal symmetry from powder X-ray diffraction, then rescored four previously released vision transformers, their own 1D convolutional network and six further architectures they trained.
spectrum · one line per step, placed by what the step does · bright lines used AI
InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification
arXiv, 2026
doi:10.48550/arxiv.2611.01525 · record aix-00302 v2 · checked 2026-11-10
- AI was for
- Classification
- Model family
- Transformer, Convolutional neural network, Recurrent neural network
- Checked by
- Held-out473 tested
- Code
- available
The finding the paper is about came from the AI.
What this research was about
When X-rays are fired at a powdered solid, the crystals scatter them into a pattern of peaks at particular angles. That pattern carries clues about how the atoms are arranged, including the symmetry of the repeating unit. Crystallographers sort all possible symmetries into 230 space groups, and a coarser scheme of 99 extinction groups, which collects together space groups that suppress the same peaks and so cannot be told apart from peak positions alone. Reading symmetry off a measured pattern is awkward: real scans carry instrument quirks, preferred crystal alignment and overlapping peaks, and some symmetries are far more common in nature than others.
That last point makes scoring a classifier tricky. A model that simply guessed the commonest groups every time can look accurate. The authors set out to pin down what a reported accuracy figure on this task actually means. They assembled a machine-readable table linking the 230 space groups to the 99 extinction groups, wrote the rules for building test sets as code, and froze three evaluation sets drawn from experimental archives: RRUFF-473, RRUFF-325 and opXRD-98. They then specified a reporting protocol, including guessing baselines that ignore the input, a shuffled-label test, and an error measure that asks how far a wrong answer sits from the right one in the hierarchy of crystal symmetries.
Where AI came in
The things being measured here are neural networks. Four vision transformer checkpoints from the authors' earlier work were run as released, a small one-dimensional residual convolutional network was trained from scratch as a control, and six further designs, including convolutional networks, two recurrent networks and transformers, were trained under a public recipe on simulated diffraction patterns. A matched pair of transformers was also trained, one predicting all 230 space groups and one predicting the 99 extinction groups, to see what the choice of target costs. Training used simulated patterns, since labelled experimental scans are scarce.
Every model was then run over the frozen experimental benchmarks, with 230-way outputs folded into the 99 extinction groups, and the resulting per-pattern predictions were scored. The networks stand in for the classical route to symmetry, in which a crystallographer fits a model of the crystal to the measured pattern by hand. Classical fitting still appears, but as a way of grading how messy each experimental scan is, so accuracy can be reported separately for easy and awkward patterns rather than as one number. Every reported result is a statement about what these models predicted.
Written by AIxSci from the checked record below, to give context for readers outside the field. It is not part of the record.
The work
Technical · from the record
InformedXRD is an evaluation framework for machine-learning classification of crystal symmetry from powder X-ray diffraction patterns. It combines a machine-readable mapping from the 230 space groups to 99 extinction groups, benchmark-construction rules specified as code that yield the frozen RRUFF-473, RRUFF-325 and opXRD-98 sets, and a reporting protocol of input-blind frequency baselines, a label-permutation test, a hierarchy-aware error metric on a condensed translationengleiche subgroup graph, and stratified reporting. The authors rescored four previously released ViT checkpoints, an in-house 1D residual CNN, and six architectures they trained under the XRDBench recipe. Reported findings include that exact McNemar tests favour the preferred-orientation-only ViT checkpoint at Top-1 and the final calibrated checkpoint at Top-5 on RRUFF-325, that an input-blind predictor using a benchmark's own label histogram would match or exceed the final calibrated checkpoint's aggregate Top-k while a label-permutation test still detects a small pattern-specific association, and that reweighting strata to a common extinction-group distribution removes about 45% of the recoverable-versus-catastrophic gap.
How AI was used
Four ViT extinction-group checkpoints released in the authors' earlier work were run with the parameters they were released with. A 1D residual CNN control with 646,596 parameters was trained from scratch with AdamW (learning rate 1e-3, weight decay 1e-4, batch size 128, cross-entropy loss, seed 1337) on a 2.35M-pattern RRUFF-conditioned synthetic corpus of 1,878,370 training and 237,056 validation patterns on an 8501-point 5–90° 2θ grid, and its lineage was resumed for 40 further epochs with checkpoint selection on held-out synthetic validation loss. Six architectures (CNN10, CNN11, PatchTST, BiLSTM, BiGRU and a generic Transformer) were trained under the public XRDBench recipe on the unchanged SimXRD train_1 contract (597,845 patterns, 230 space-group targets, 10–80° at 0.02°, 3,501 points per pattern), and a matched pair of 230-way space-group and 99-way extinction-group ViTs was trained on a two-million-pattern synthetic corpus sharing split, backbone, initialisation, optimiser, augmentations and budget. All models were then run in inference over the experimental benchmark patterns, with 230-way probabilities summed into the 99-way extinction-group space, and their per-pattern predictions were scored with Top-k accuracy, exact McNemar and paired bootstrap tests, deployable and oracle prior-only frequency baselines, a 10,000-permutation label-permutation test, hop distance and direction on the condensed t-subgroup DAG against uniform, model-matched and frequency-matched nulls, and per-stratum reporting with entropy reweighting.
The shape of the work
Structural · the record, drawn
Build canonicalized extinction-group label space
Cleaning, filtering, normalising or labelling data already obtained.
we fetch and parse that table, expand its range and list notation into explicit space-group numberswhere the paper describes this · verbatim
Construct frozen benchmarks from experimental archives
Cleaning, filtering, normalising or labelling data already obtained.
Family-coherence checks reduce the 1,981-scan Cu-K α pool to RRUFF-473where the paper describes this · verbatim
Train case-study classifiers on simulated patterns
Fitting model parameters, including fine-tuning an existing model.
six independently developed architectures—CNN10, CNN11, PatchTST, BiLSTM, BiGRU, and a generic Transformer—trained under the XRDBench recipe of SimXRDwhere the paper describes this · verbatim
Rescore all models on the benchmarks in extinction-group space
Running a trained model over new data to predict, classify or score.
collapsed into the 99-way space with the §3 mapping and scored on identical RRUFF-473 and RRUFF-325 inputswhere the paper describes this · verbatim
Score against prior-only baselines and label-permutation test
Testing outputs against ground truth.
ranking the extinction groups by their training frequency and predicting that same fixed Top- k list for every patternwhere the paper describes this · verbatim
Characterise error distance and direction on the subgroup DAG
Extracting understanding from model behaviour.
For each wrong Top-1 prediction we compute three quantities: distance, direction, and locality.where the paper describes this · verbatim
Stratify accuracy and reweight for label-space entropy
Testing outputs against ground truth.
Stratified evaluation partitions the benchmark along both axes and reports per-stratum accuracy alongside the aggregate.where the paper describes this · verbatim
What the record says
Technical · every part carries its own basis
+ in the paper~ our reading− not reported
How to read the quotations. A quotation shows where the paper describes something. It does not quote every value beside it: one passage locates a part of the work, and values without their own quotation are our reading of that passage.
The paper's findings are statements about model predictions: every reported result comes from running trained classifiers over the benchmarks and scoring their outputs, so the results could not exist without the models. The paper itself contributes evaluation infrastructure rather than a model, so 'analysis' is a defensible alternative reading.
an in-house 1D residual CNN, and six architectures trained under the XRDBench recipewhere the paper describes this · verbatim
Both are fixed evaluation sets, not training corporawhere the paper describes this · verbatim
The dataset repository is released under CC-BY-4.0 and the checkpoint and code repositories under the MIT licensewhere the paper describes this · verbatim
What this paper did not report
Technical · absence is published deliberately
- Version of ViT extinction-group checkpoints of Baggett et al. (four variants: no-preferred-orientation, preferred-orientation-only, final mixed-data calibrated, legacy uncalibrated)Which version of the model was used is not stated.
- Version of 1D residual CNN control (epoch-6 checkpoint; lineage also resumed to epoch 46)Which version of the model was used is not stated.
- Version of CNN10 (XRDBench recipe)Which version of the model was used is not stated.
- Version of CNN11 (XRDBench recipe)Which version of the model was used is not stated.
- Version of PatchTST (XRDBench recipe)Which version of the model was used is not stated.
- Version of BiLSTM (XRDBench recipe)Which version of the model was used is not stated.
- Version of BiGRU (XRDBench recipe)Which version of the model was used is not stated.
- Version of Generic Transformer (XRDBench recipe)Which version of the model was used is not stated.
- Version of 230-way space-group ViT (matched target-choice comparison arm)Which version of the model was used is not stated.
- What step 3 replacedThe paper gives no basis for what the AI stood in for.
- What step 4 replacedThe paper gives no basis for what the AI stood in for.
About this article
Record aix-00302, version 2, checked by a person on 2026-11-10. The record describes the paper; it does not assess whether the paper's findings are right. The paper is published under CC-BY; quotations are at most 25 words. How we work · Report an error
Cite this record: AIxSci (2026). A test suite for judging how well AI reads crystal symmetry from X-ray patterns. Record aix-00302, version 2, checked 2026-11-10. https://aixsci.org/articles/aix-00302/