Report

Crystal symmetry determination in electron diffraction using machine learning

See allHide authors and affiliations

Science  31 Jan 2020:
Vol. 367, Issue 6477, pp. 564-568
DOI: 10.1126/science.aay3062

Speedy crystallography

Electron backscatter diffraction is one standard technique for determining crystal structure, typically of materials or geological samples. However, this method requires structural guesses and user input that are often time consuming or incorrect. Kaufmann et al. developed a general methodology using a convolutional neural network that automatically determines the crystal structure quickly and with high accuracy. After the network is exposed to a training set, it can identify the crystal structure without any additional input most of the time, providing a method for eliminating some of the guesswork from crystal structure determination.

Science, this issue p. 564

Abstract

Electron backscatter diffraction (EBSD) is one of the primary tools for crystal structure determination. However, this method requires human input to select potential phases for Hough-based or dictionary pattern matching and is not well suited for phase identification. Automated phase identification is the first step in making EBSD into a high-throughput technique. We used a machine learning–based approach and developed a general methodology for rapid and autonomous identification of the crystal symmetry from EBSD patterns. We evaluated our algorithm with diffraction patterns from materials outside the training set. The neural network assigned importance to the same symmetry features that a crystallographer would use for structure identification.

Identifying structure is a crucial step for the analysis of proteins (13), micro- (4, 5) and macromolecules (6), pharmaceuticals (7), geological specimens (8), synthetic materials (911), and many other types of materials. Crystal structure plays an important role in the material properties exhibited (12, 13). Determining the crystal symmetry, lattice parameters, and atom positions of the crystal phases is a challenging task, especially for low-symmetry phases and multiphase materials. The most common techniques involve either x-ray diffraction (XRD) or transmission electron microscopy (TEM)–based convergent beam electron diffraction (CBED) (1416). XRD requires only a sample powder or a polished bulk sample and in most cases only a few hours to collect diffraction intensities over a range of angles. Researchers must refine diffraction patterns to match the experimentally collected pattern to one in a database or from a theoretical model. This process has drawbacks because structural misclassification can occur as a result of lattice parameter shifts, overlapping XRD peaks in multiphase samples, texture effects, and the matching thresholds set by researchers. TEM studies that use CBED are more precise than XRD in their ability to pinpoint the location of individual crystals, produce singular diffraction patterns for a given phase, and capture subtle symmetry information. However, sample preparation, data collection rates, data analysis rates, and the requirement of substantial operator experience limit the throughput for CBED-based studies (1719).

A scanning electron microscope (SEM) equipped with an electron backscatter diffraction (EBSD) system has become important for the characterization of crystalline materials and geological samples (8). Nishikawa and Kikuchi discovered EBSD patterns in 1928 (20). Early research by Alam et al. (21), Venables et al. (22), and Dingley (23) led to the emergence of commercial EBSD systems. The development of fully automated image analysis methods occurred in the early 1990s (24, 25). After the introduction of automated EBSD, commercial software and hardware have evolved to capture more than 3000 patterns per second, which expands the applicability of the technique to assist researchers with more complex problems (26). For example, the high-throughput capability of a modern EBSD system enables determination of fine-scale grain structures, sample texture, point-to-point crystal orientation, residual stress or strain, geometrically necessary dislocation densities, and other information (2731). The relative ease of sample preparation compared with TEM samples and the larger sample area analysis in less time makes SEM-EBSD an attractive technique for studies of location-specific orientation with high precision (~2°), high misorientation resolution (0.2°), and high spatial resolution (~40 nm) (32). One of the most common applications of EBSD in multiphase samples is phase differentiation along with orientation determination. A user selects the phases presumed to be in the sample, and a program finds the best-fit phase and orientation to the diffraction pattern (33). Selected libraries of simulated diffraction patterns of phases can be used in a dictionary indexing approach to assist with phase differentiation, including when working with deformed or fine-grained materials (34). Phase identification is possible when combined with other analytical techniques such as energy-dispersive x-ray spectroscopy (EDS) or wavelength-dispersive x-ray spectroscopy (WDS) (33, 35, 36). This requires that the chemical and structural information of the phase exists in a theoretical model or crystal database, such as the Inorganic Crystal Structure Database. A method was developed to determine the crystal structure using EBSD without EDS data, but this requires hand-drawn lines to be overlaid with a high degree of accuracy on individual Kikuchi bands (35, 37). This method is slow and tedious, as it requires manual annotation of each individual pattern. In general, EBSD has been limited to elucidating the orientation of user-defined crystal structures.

Recently, the materials science field has begun to embrace the big data revolution (38). Researchers have shown the ability to predict new compositions for bulk metallic glasses (39), shape-memory alloys (40), Heusler compounds (41, 42), and ultra-incompressible superhard materials (43). Other groups are developing machine-learning methods to establish structure-property linkages (4446) or to predict the crystal stability of new materials (47). Holm and colleagues (48, 49) have demonstrated the classification of optical microscopy images into one of seven groups with greater than 80% accuracy (48), as well as microconstituent segmentation using the PixelNet convolutional neural network (CNN) architecture trained on manually annotated micrographs of ultrahigh-carbon steel (49). These machine learning–driven analysis techniques represent important developments in the materials science toolbox. Previous studies have attempted crystal symmetry identification using deep neural networks and TEM diffraction; however, the developed model’s practical use is hindered by the choice to use images simulated in RGB color, whereas real TEM diffraction patterns are captured in grayscale (50). Another study used full XRD pattern images for single-phase materials (51). These techniques only provide point (TEM) or global (XRD) information about the sample; by contrast, EBSD’s mapping capabilities can provide spatially relevant crystallographic information across many length scales. Here, we demonstrate a hybrid methodology, EBSD coupled with a machine-learning algorithm, to identify the Bravais lattice or space group of a bulk sample from diffraction patterns. The trained machine-learning model is subsequently applied to a distinct set of materials it was not trained on, but which contain the same crystal symmetry, and it identifies the correct Bravais lattice or space group with a high degree of accuracy.

We used two CNNs in this work. The two image classification model architectures are ResNet50 (52) and Xception (53) (Fig. 1). We started by constructing a convolution layer, where a learnable filter is convolved across the image. We computed the scalar product between the filter and the input at every position, or “patch,” to form a feature map. Next, we sequentially stacked a series of alternating convolutional and pooling layers. We organized the feature maps with the units in a convolutional layer and connected each feature map to local patches in the previous layer through a set of weights called a filter bank. All units in a feature map shared the same filter banks (also called kernels), whereas different feature maps in a convolutional layer used different filter banks. We placed pooling layers after convolutional layers to downsample the feature maps. This produced coarse-grained representations and spatial information about the features in the data. The trained layers of feature detection nodes are “learned” from the data as the algorithm finds motifs encoding the underlying crystallographic symmetry present in the diffraction patterns.

Fig. 1 Illustration of the inner workings of a convolutional neural network.

Convolutional neural networks (CNNs) are composed of a series of alternating convolutional and pooling layers. Each convolutional layer extracts features from its preceding layer, using filters (or kernels) learned from training the model, to form feature maps. These feature maps are then downsampled by a pooling layer to exploit data locality. A traditional dense neural network, a simple type of classification network, is placed as the last layer of the CNN, where the probability that the input diffraction pattern belongs to a given class (e.g., Bravais lattice or space group) is computed.

We found that both ResNet50 and Xception (53) CNNs performed similarly well at classifying EBSD patterns. We applied the trained model to diffraction patterns that were “new” to the algorithm. This means that the patterns were not part of the training set but rather a random mix of orientations that may or may not be similar to the training patterns. Both the ResNet50 (Fig. 2) and Xception (fig. S1) architectures correctly classified nearly 300,000 diffraction patterns with >90% overall accuracy for each architecture. Specifically, this means that no user input was required for the algorithm to identify which of the 14 Bravais lattices each individual EBSP belonged to. The main crystal structure misclassification was jadeite, a monoclinic mineral often assigned to structures containing the same symmetry elements (fig. S1, A and C). This specific misclassification resulted in an overall decrease of the algorithm’s performance. An in-depth analysis of this misclassification type was performed to understand the cause. The model displayed much higher accuracy on all other materials, typically greater than 95% for individual materials.

Fig. 2 Confusion matrix displaying the ResNet50 algorithm’s classification results.

(A) The trained algorithm classifies a second set of diffraction patterns from 14 of the materials. The diagonal (blue shaded boxes) represents the successful matching of the CNN predictions to the true Bravais lattices of the sample. (B) The algorithm classifies electron backscatter diffraction patterns collected from materials not used during training of the model. Correct classification is identified by the green squares instead of along the diagonal.

We collected 50,000 EBSD patterns from nine completely different materials for blind testing of our algorithms’ crystal symmetry identification. Each architecture correctly classified the Bravais lattice of the unknown material with 93.5% (Fig. 2B) and 91.2% (fig. S2) overall accuracy for ResNet50 and Xception, respectively. The base-centered monoclinic crystal structure has a propensity to be incorrectly classified as primitive orthorhombic or rhombohedral. The base-centered monoclinic, primitive orthorhombic, and rhombohedral Bravais lattices used in training belong to the 2/m, mmm, and 3¯m point groups, respectively. The 2/m and mmm point groups each only have two-fold axis symmetry, mirror plane symmetry, and inversion center symmetry in different multiplicity (table S1). The rhombohedral 3¯m point group shares these same symmetry elements, with the addition of one three-fold axis symmetry. Of the misclassification to these two point groups, the model displays a lesser degree of misclassification to the 3¯m point group containing the extra symmetry element, as expected for a well-fit model. This sort of misclassification event presents itself as a potential source of error, especially in a low-symmetry phase, where one pattern from the Kikuchi sphere may not contain enough symmetry elements for the best possible classification.

We also used our method to classify the small changes in atomic arrangement that distinguish space groups within the 4/m, 3¯, 2/m (cubic) point group (fig. S3) at a rate of 1 pattern per second. We found that the ResNet50 and Xception algorithms only misclassified a small portion of our as-collected dataset between the selected cubic space groups.

Walking through a specific example of feature identification by the algorithm helps us to understand how it arrives at a correct classification. We start with diffraction patterns of nickel and aluminum with similar crystallographic orientation (Fig. 3). The importance of features in each image is determined by the learned filter banks in the algorithm. The importance of local regions in the image is elucidated using the trained neural network architecture and a set of tools called Grad-CAM (54). After the algorithm computes the “importance” of these local regions, Grad-CAM maps the normalized weights from 0 (dark blue) to 1 (dark red). These heat maps are similar for nickel and aluminum and show an intense interest of the network in symmetry located at the zone axes. The regions of greatest interest are the [11¯2] and [112] zone axes (two-fold symmetry). The machine-learning algorithm couples this information with the presence of the [001] (four-fold symmetry) and [01¯3] (two-fold symmetry) zone axes and their spatial relationship, owing to pooling layers, to correctly identify the Bravais lattice as face-centered cubic. We observed a similar interest in information nearest the zone axes for the other materials.

Fig. 3 Visualizing the features used for classification of a diffraction pattern as face-centered cubic (fcc).

Electron backscatter diffraction patterns from nickel (Ni) and aluminum (Al) were selected from nearly identical orientations. In the diffraction patterns, four of the zone axes present in each are labeled. The corresponding heat maps display the importance of information in the image for correctly classifying it as fcc. Note that for each of these two images, the symmetry information near the [11¯2] zone axis produces the highest activation, followed by the [112] zone axis and the symmetry shared by the [001] and [01¯3] zone axes.

We determined heat maps for the 28 materials we used in the training set (fig. S4). This allowed us to investigate where the algorithm has difficulty with identifications. We used Grad-CAM to investigate the misidentification of diopside (fig. S5). The base-centered monoclinic and primitive orthorhombic class both result in similar activations, with interest centered around the only “x-fold” symmetry present in diopside, the two-fold symmetry [112¯] zone axis. Because the base-centered monoclinic and primitive orthorhombic structures differ only on the number of two-fold axes and do not possess higher-symmetry elements, the algorithm has difficulty distinguishing between the structures. We observed that the area of greatest interest is not always centered around the bright spot of a zone axis, as for Cr3Si or Sn, and instead favors the side with other zone axes nearby in the diffraction pattern.

Our algorithm reduced the amount of prior sample knowledge required for crystal structure identification. A common approach for crystal identification is to run the diffraction images through a Hough transform, which helps to extract diffraction maxima at Kikuchi band intersections. This method can lead to misclassification of similar crystal structures that have similar diffraction maxima (5557). In contrast, our algorithm autonomously uses all the information in each diffraction pattern. To demonstrate how this helps with a multiphase sample, we used rutilated quartz (Fig. 4), which contains a phase that was not in our training set. Our machine learning–generated phase map is nearly identical to the one generated by the Hough-transform method. Of the seven errors, five are located where the traditional method could not index the structure.

Fig. 4 Comparison of phase-mapping techniques.

(A) Phase map generated by traditional Hough-transform EBSD, where the user had to select quartz and rutile as the two specific phases present in the sample. Black pixels could not be identified. (B) Electron image of the region of the sample from which the diffraction patterns were collected. The quartz appears recessed and rutile emerges above the surface. (C) Phase map generated via machine-learning determination of the Bravais lattice for each diffraction pattern. Scale bar in (A), 100 μm.

Our methodology enables high-throughput and autonomous determination of crystal symmetry in electron backscatter diffraction. We found that the CNN identifies specific features resulting from unique crystal symmetry operations within the diffraction pattern images. The method can potentially be expanded to encompass a multi-tiered model to determine the complete crystal structure. Improvements building on our methodology include neural network architectures specifically designed for specific multiphase samples or through incorporating other data (e.g., phase chemistry) into the algorithm. We believe that a wide range of research areas including pharmacology, structural biology, and geology would benefit by using automated algorithms that reduce the amount of time required for structural identification.

Supplementary Materials

science.sciencemag.org/content/367/6477/564/suppl/DC1

Materials and Methods

Supplementary Text

Table S1

Figs. S1 to S10

References (5865)

References and Notes

Acknowledgments: We thank E. Chin of the Scripps Institute of Oceanography and her students for their assistance collecting and identifying mineral and geological samples. Funding: Supported by the U.S. Department of Defense (DoD) [through the National Defense Science and Engineering Graduate Fellowship (NDSEG) Program] and the ARCS Foundation, San Diego Chapter (K.K.); the Joint DoD/Department of Energy Munitions Technology Development Program and the Dynamic Materials Science Campaign at LANL (C.Z.); and the Oerlikon Group (K.S.V.). Author contributions: K.K. assisted in developing the idea, performed the bulk of the experimental work, worked on later versions of the neural network python code, and prepared the initial draft of the manuscript and figures; C.Z. assisted with the development of the idea into a scientific study, contributed his knowledge of EBSD, assisted with the figures, and developed MATLAB code for phase-mapping the machine-learning predictions and pattern database management; A.S.R. developed the initial python code for implementing the neural networks; A.S.R. and D.M. managed the deployment of Grad-CAM tools and assisted in the analysis of the results from the deep learning models; T.J.H. assisted K.K. with materials selection, fabrication, and processing; T.J.H. helped focus the research direction; E.M. assisted K.K. with materials fabrication, processing, and analysis; K.S.V. led the development of the idea, guided the focus of the project, and reviewed and revised the manuscript; and all authors participated in analyzing and interpreting the final data and contributed to the discussions and revisions of the manuscript. Competing interests: The authors declare no competing interests. Data and materials availability: All data and models generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request. The python code for implementing these models with Keras and Tensorflow is available at https://github.com/krkaufma/Electron-Diffraction-CNN, Zenodo (DOI: 10.5281/zenodo.3564937) and from the corresponding author upon request. The diffraction patterns analyzed during the current study have not yet been deposited in a publicly available repository because of the sheer size of the library (nearly 3 terabytes). This is an ongoing effort and until then, they are available upon reasonable request.

Stay Connected to Science

Navigate This Article