Speedy crystallography
Electron backscatter diffraction is one standard technique for determining crystal structure, typically of materials or geological samples. However, this method requires structural guesses and user input that are often time consuming or incorrect. Kaufmann et al. developed a general methodology using a convolutional neural network that automatically determines the crystal structure quickly and with high accuracy. After the network is exposed to a training set, it can identify the crystal structure without any additional input most of the time, providing a method for eliminating some of the guesswork from crystal structure determination.
Science, this issue p. 564
Abstract
Electron backscatter diffraction (EBSD) is one of the primary tools for crystal structure determination. However, this method requires human input to select potential phases for Hough-based or dictionary pattern matching and is not well suited for phase identification. Automated phase identification is the first step in making EBSD into a high-throughput technique. We used a machine learning–based approach and developed a general methodology for rapid and autonomous identification of the crystal symmetry from EBSD patterns. We evaluated our algorithm with diffraction patterns from materials outside the training set. The neural network assigned importance to the same symmetry features that a crystallographer would use for structure identification.
Identifying structure is a crucial step for the analysis of proteins (1–3), micro- (4, 5) and macromolecules (6), pharmaceuticals (7), geological specimens (8), synthetic materials (9–11), and many other types of materials. Crystal structure plays an important role in the material properties exhibited (12, 13). Determining the crystal symmetry, lattice parameters, and atom positions of the crystal phases is a challenging task, especially for low-symmetry phases and multiphase materials. The most common techniques involve either x-ray diffraction (XRD) or transmission electron microscopy (TEM)–based convergent beam electron diffraction (CBED) (14–16). XRD requires only a sample powder or a polished bulk sample and in most cases only a few hours to collect diffraction intensities over a range of angles. Researchers must refine diffraction patterns to match the experimentally collected pattern to one in a database or from a theoretical model. This process has drawbacks because structural misclassification can occur as a result of lattice parameter shifts, overlapping XRD peaks in multiphase samples, texture effects, and the matching thresholds set by researchers. TEM studies that use CBED are more precise than XRD in their ability to pinpoint the location of individual crystals, produce singular diffraction patterns for a given phase, and capture subtle symmetry information. However, sample preparation, data collection rates, data analysis rates, and the requirement of substantial operator experience limit the throughput for CBED-based studies (17–19).
A scanning electron microscope (SEM) equipped with an electron backscatter diffraction (EBSD) system has become important for the characterization of crystalline materials and geological samples (8). Nishikawa and Kikuchi discovered EBSD patterns in 1928 (20). Early research by Alam et al. (21), Venables et al. (22), and Dingley (23) led to the emergence of commercial EBSD systems. The development of fully automated image analysis methods occurred in the early 1990s (24, 25). After the introduction of automated EBSD, commercial software and hardware have evolved to capture more than 3000 patterns per second, which expands the applicability of the technique to assist researchers with more complex problems (26). For example, the high-throughput capability of a modern EBSD system enables determination of fine-scale grain structures, sample texture, point-to-point crystal orientation, residual stress or strain, geometrically necessary dislocation densities, and other information (27–31). The relative ease of sample preparation compared with TEM samples and the larger sample area analysis in less time makes SEM-EBSD an attractive technique for studies of location-specific orientation with high precision (~2°), high misorientation resolution (0.2°), and high spatial resolution (~40 nm) (32). One of the most common applications of EBSD in multiphase samples is phase differentiation along with orientation determination. A user selects the phases presumed to be in the sample, and a program finds the best-fit phase and orientation to the diffraction pattern (33). Selected libraries of simulated diffraction patterns of phases can be used in a dictionary indexing approach to assist with phase differentiation, including when working with deformed or fine-grained materials (34). Phase identification is possible when combined with other analytical techniques such as energy-dispersive x-ray spectroscopy (EDS) or wavelength-dispersive x-ray spectroscopy (WDS) (33, 35, 36). This requires that the chemical and structural information of the phase exists in a theoretical model or crystal database, such as the Inorganic Crystal Structure Database. A method was developed to determine the crystal structure using EBSD without EDS data, but this requires hand-drawn lines to be overlaid with a high degree of accuracy on individual Kikuchi bands (35, 37). This method is slow and tedious, as it requires manual annotation of each individual pattern. In general, EBSD has been limited to elucidating the orientation of user-defined crystal structures.
Recently, the materials science field has begun to embrace the big data revolution (38). Researchers have shown the ability to predict new compositions for bulk metallic glasses (39), shape-memory alloys (40), Heusler compounds (41, 42), and ultra-incompressible superhard materials (43). Other groups are developing machine-learning methods to establish structure-property linkages (44–46) or to predict the crystal stability of new materials (47). Holm and colleagues (48, 49) have demonstrated the classification of optical microscopy images into one of seven groups with greater than 80% accuracy (48), as well as microconstituent segmentation using the PixelNet convolutional neural network (CNN) architecture trained on manually annotated micrographs of ultrahigh-carbon steel (49). These machine learning–driven analysis techniques represent important developments in the materials science toolbox. Previous studies have attempted crystal symmetry identification using deep neural networks and TEM diffraction; however, the developed model’s practical use is hindered by the choice to use images simulated in RGB color, whereas real TEM diffraction patterns are captured in grayscale (50). Another study used full XRD pattern images for single-phase materials (51). These techniques only provide point (TEM) or global (XRD) information about the sample; by contrast, EBSD’s mapping capabilities can provide spatially relevant crystallographic information across many length scales. Here, we demonstrate a hybrid methodology, EBSD coupled with a machine-learning algorithm, to identify the Bravais lattice or space group of a bulk sample from diffraction patterns. The trained machine-learning model is subsequently applied to a distinct set of materials it was not trained on, but which contain the same crystal symmetry, and it identifies the correct Bravais lattice or space group with a high degree of accuracy.
We used two CNNs in this work. The two image classification model architectures are ResNet50 (52) and Xception (53) (Fig. 1). We started by constructing a convolution layer, where a learnable filter is convolved across the image. We computed the scalar product between the filter and the input at every position, or “patch,” to form a feature map. Next, we sequentially stacked a series of alternating convolutional and pooling layers. We organized the feature maps with the units in a convolutional layer and connected each feature map to local patches in the previous layer through a set of weights called a filter bank. All units in a feature map shared the same filter banks (also called kernels), whereas different feature maps in a convolutional layer used different filter banks. We placed pooling layers after convolutional layers to downsample the feature maps. This produced coarse-grained representations and spatial information about the features in the data. The trained layers of feature detection nodes are “learned” from the data as the algorithm finds motifs encoding the underlying crystallographic symmetry present in the diffraction patterns.
Convolutional neural networks (CNNs) are composed of a series of alternating convolutional and pooling layers. Each convolutional layer extracts features from its preceding layer, using filters (or kernels) learned from training the model, to form feature maps. These feature maps are then downsampled by a pooling layer to exploit data locality. A traditional dense neural network, a simple type of classification network, is placed as the last layer of the CNN, where the probability that the input diffraction pattern belongs to a given class (e.g., Bravais lattice or space group) is computed.
We found that both ResNet50 and Xception (53) CNNs performed similarly well at classifying EBSD patterns. We applied the trained model to diffraction patterns that were “new” to the algorithm. This means that the patterns were not part of the training set but rather a random mix of orientations that may or may not be similar to the training patterns. Both the ResNet50 (Fig. 2) and Xception (fig. S1) architectures correctly classified nearly 300,000 diffraction patterns with >90% overall accuracy for each architecture. Specifically, this means that no user input was required for the algorithm to identify which of the 14 Bravais lattices each individual EBSP belonged to. The main crystal structure misclassification was jadeite, a monoclinic mineral often assigned to structures containing the same symmetry elements (fig. S1, A and C). This specific misclassification resulted in an overall decrease of the algorithm’s performance. An in-depth analysis of this misclassification type was performed to understand the cause. The model displayed much higher accuracy on all other materials, typically greater than 95% for individual materials.
(A) The trained algorithm classifies a second set of diffraction patterns from 14 of the materials. The diagonal (blue shaded boxes) represents the successful matching of the CNN predictions to the true Bravais lattices of the sample. (B) The algorithm classifies electron backscatter diffraction patterns collected from materials not used during training of the model. Correct classification is identified by the green squares instead of along the diagonal.
We collected 50,000 EBSD patterns from nine completely different materials for blind testing of our algorithms’ crystal symmetry identification. Each architecture correctly classified the Bravais lattice of the unknown material with 93.5% (Fig. 2B) and 91.2% (fig. S2) overall accuracy for ResNet50 and Xception, respectively. The base-centered monoclinic crystal structure has a propensity to be incorrectly classified as primitive orthorhombic or rhombohedral. The base-centered monoclinic, primitive orthorhombic, and rhombohedral Bravais lattices used in training belong to the 2/m, mmm, and
We also used our method to classify the small changes in atomic arrangement that distinguish space groups within the 4/m,
Walking through a specific example of feature identification by the algorithm helps us to understand how it arrives at a correct classification. We start with diffraction patterns of nickel and aluminum with similar crystallographic orientation (Fig. 3). The importance of features in each image is determined by the learned filter banks in the algorithm. The importance of local regions in the image is elucidated using the trained neural network architecture and a set of tools called Grad-CAM (54). After the algorithm computes the “importance” of these local regions, Grad-CAM maps the normalized weights from 0 (dark blue) to 1 (dark red). These heat maps are similar for nickel and aluminum and show an intense interest of the network in symmetry located at the zone axes. The regions of greatest interest are the
Electron backscatter diffraction patterns from nickel (Ni) and aluminum (Al) were selected from nearly identical orientations. In the diffraction patterns, four of the zone axes present in each are labeled. The corresponding heat maps display the importance of information in the image for correctly classifying it as fcc. Note that for each of these two images, the symmetry information near the
We determined heat maps for the 28 materials we used in the training set (fig. S4). This allowed us to investigate where the algorithm has difficulty with identifications. We used Grad-CAM to investigate the misidentification of diopside (fig. S5). The base-centered monoclinic and primitive orthorhombic class both result in similar activations, with interest centered around the only “x-fold” symmetry present in diopside, the two-fold symmetry
Our algorithm reduced the amount of prior sample knowledge required for crystal structure identification. A common approach for crystal identification is to run the diffraction images through a Hough transform, which helps to extract diffraction maxima at Kikuchi band intersections. This method can lead to misclassification of similar crystal structures that have similar diffraction maxima (55–57). In contrast, our algorithm autonomously uses all the information in each diffraction pattern. To demonstrate how this helps with a multiphase sample, we used rutilated quartz (Fig. 4), which contains a phase that was not in our training set. Our machine learning–generated phase map is nearly identical to the one generated by the Hough-transform method. Of the seven errors, five are located where the traditional method could not index the structure.
(A) Phase map generated by traditional Hough-transform EBSD, where the user had to select quartz and rutile as the two specific phases present in the sample. Black pixels could not be identified. (B) Electron image of the region of the sample from which the diffraction patterns were collected. The quartz appears recessed and rutile emerges above the surface. (C) Phase map generated via machine-learning determination of the Bravais lattice for each diffraction pattern. Scale bar in (A), 100 μm.
Our methodology enables high-throughput and autonomous determination of crystal symmetry in electron backscatter diffraction. We found that the CNN identifies specific features resulting from unique crystal symmetry operations within the diffraction pattern images. The method can potentially be expanded to encompass a multi-tiered model to determine the complete crystal structure. Improvements building on our methodology include neural network architectures specifically designed for specific multiphase samples or through incorporating other data (e.g., phase chemistry) into the algorithm. We believe that a wide range of research areas including pharmacology, structural biology, and geology would benefit by using automated algorithms that reduce the amount of time required for structural identification.
Supplementary Materials
science.sciencemag.org/content/367/6477/564/suppl/DC1
Materials and Methods
Supplementary Text
Table S1
Figs. S1 to S10
This is an article distributed under the terms of the Science Journals Default License.
