## Abstract

The recently discovered three-dimensional or bulk topological insulators are expected to exhibit exotic quantum phenomena. It is believed that a trivial insulator can be twisted into a topological state by modulating the spin-orbit interaction or the crystal lattice, driving the system through a topological quantum phase transition. By directly measuring the topological quantum numbers and invariants, we report the observation of a phase transition in a tunable spin-orbit system, BiTl(S_{1–δ}Se_{δ})_{2}, in which the topological state formation is visualized. In the topological state, vortex-like polarization states are observed to exhibit three-dimensional vectorial textures, which collectively feature a chirality transition as the spin momentum–locked electrons on the surface go through the zero carrier density point. Such phase transition and texture inversion can be the physical basis for observing fractional charge (±e/2) and other fractional topological phenomena.

Topological insulators in three dimensions are nonmagnetic insulators with novel surface states that are a consequence of the nontrivial topology of electronic wave functions in the bulk of the materials (*1*–*5*). Their experimental discoveries in Bi-Sb semiconductors, and in Bi_{2}Se_{3} and related materials, have led to the exploration of topological quantum phenomena at modest temperatures and without the requirement of any applied magnetic field (*6*–*16*). Current theoretical proposals have suggested such intriguing possibilities as the realization of fractional quasi-particles and the exploitation of the role of spin momentum locking, which leads to quantum phases (such as topological Berry’s phases), in transport (*17*–*24*).

Strong topological insulators are distinguished from ordinary insulators by a finite topological quantum number or an invariant (ν_{0}) or, equivalently, an axion angle parameter (θ). The value of ν_{0} or θ depends on the parity eigenvalues of the wave functions in time reversal–symmetric materials (*1*, *2*) and can be determined from spin texture details of the spin-orbit surface states that form when the bulk is terminated, as demonstrated by Hsieh *et al.* (*6*–*9*). In particular, a ν_{0} = 1 = θ/π topology requires the terminated surface to have a Fermi surface that supports a nonzero Berry’s phase (an odd as opposed to an even multiple of π = θ = axion angle), which is not realizable in an ordinary spin-orbit or trivial (ν_{0} = 0 = θ) insulator. It is believed that a trivial insulator can be twisted into a ν_{0} = 1 = θ/π topological state by signaling the appearance of unusual spin vortex–like [or skyrmion-like (*1*)] arrangements via increasing spin-orbit interaction or by modulating the lattice parameters, thereby driving the system through a topological quantum phase transition (*1*, *2*). However, the topological insulators Bi_{2}X_{3} (X = Se,Te) cannot be tuned out from a trivial insulator version without a structural phase transition; in the Bi-Sb semiconductors, the topological phase transition is masked by an intervening band [the “H” band (*1*)] and the lack of gating control necessary for its observation.

We demonstrate the existence of a topological phase transition in TlBi(S_{1–δ}Se_{δ})_{2}, which, as we show, is a fully tunable topological analog of Bi_{2}Se_{3}. By effectively varying the sulfur/selenium ratio (δ), both the spin-orbit strength and lattice parameters are effectively tuned (*25*). Because the topological order in TlBi(S_{1–δ}Se_{δ})_{2} originates from Bi and Se atoms, and to illustrate an analogy with the known topological insulator Bi_{2}Se_{3} (*6*, *8*, *11*), we rewrite the formula as BiTl(S_{1–δ}Se_{δ})_{2}. Figure 1A presents systematic photoemission measurements of electronic states that lie between a pair of time reversal–invariant points or Kramers’ points (_{1–δ}Se_{δ})_{2}. As the selenium concentration is increased, the low-lying bands, separated by a gap of energy 0.15 eV at δ = 0.0, are observed to approach each other; the gap decreases to less than 0.05 eV at δ = 0.4. Both bands demarcating the gap show three-dimensional (3D) dispersion where binding energies vary with momentum perpendicular to the surface, *k _{z}* [as probed via varying incident photon energy (

*25*)], and roughly correspond to the expected position for the valence and conduction bands. The absence of surface states within the bulk gap suggests that the compound is topologically trivial (ν

_{0}= 0 = θ) within a composition range of δ = 0.0 to 0.4. Starting from δ = 0.6, a linearly dispersive band connecting the bulk conduction and valence bands emerges, which threads across the bulk band gap. Incident photon energy modulation studies support the assignment of these Dirac-like bands as originating from the surface (

*25*). Moreover, the band continua in the composition range of δ = 0.0 to δ = 0.4 are degenerate, whereas the Dirac-like bands at δ = 0.6 and beyond are spin-polarized (see Figs. 2, 3, and 4).

The system enters a topologically nontrivial phase upon the occurrence of an electronic phase transition between δ = 0.4 and δ = 0.6 at temperatures below 15 K. Although the system approaches the transition from the conventional or no-surface-state side (δ = 0.4), both energy dispersion and Fermi surface mapping (Fig. 1, A and B, for δ = 0.4) show that at the outer boundary of the bulk conduction band continuum that corresponds to the loci where the Dirac surface states would eventually develop, the spectral weight becomes much more intense; however, the surface remains gapped at δ = 0.4, which suggests that the material is still on the trivial side. Finer control of bulk compositional variation does not allow us to locate a precise value for the transition; this could also be a consequence of an intrinsically broader topological transition. A critical signature of a topological phase transition is that the material turns into an indirect bulk band gap material, as conjectured previously (*1*). As δ varies from 0.0 to 1.0 (Fig. 1C), the dispersion of the valence band evolves from a single-peak to a dual-peak shape with a “dip” at the *k* = 0); the δ = 0.0 compound features a direct band gap in its bulk, whereas the δ = 1.0 compound has a slightly indirect gap. The overall experimental evolution of the spin-orbit ground state is shown in Fig. 2A.

Now we systematically study the end product of the transition, BiTl(S_{0}Se_{1})_{2}, and explore its spin and polarization properties far away from the Dirac node where nonlinear spin-orbit terms are also important. Such terms correspond to analogs of cubic and higher-order Dresselhaus effects arising from the symmetry of the crystal potential. Without these effects, the surface states form a cone that is isotropic in momentum space (*k _{x}*,

*k*) and hence generates a circular Fermi contour (

_{y}*2*). Spins tangentially arranged on such a circular Fermi surface lead to a Berry’s phase of π, which is also a measure of the topological invariant and the axion angle (ν

_{0}= 1 = θ/π) of the system (

*1*,

*4*,

*6*). However, the details of the symmetry and bulk crystal potential can strongly deform the surface states from that of a circle. We explored the Fermi contour of BiTl(S

_{0}Se

_{1})

_{2}over a large energy range. This sample features an almost perfectly hexagonal Fermi surface at its native (as-grown) Fermi level (binding energy

*E*

_{B}= 0.01 eV), whereas the constant energy contour reverts to isotropic circular shapes when approaching the Dirac node (

*E*

_{B}= 0.25 eV and

*E*

_{B}= 0.50 eV). A more extreme example is n-type bulk doped Bi

_{2}Te

_{3}, a topological insulator that features a highly warped concave-in snowflake-shaped Fermi surface because of its large

*k*

_{F}and a small band gap (

*25*). To understand the relationship between these Dirac cone deformations and the topological invariants and axion angles (ν

_{0}and θ), we used spin–angle-resolved photoemission spectroscopy (spin-ARPES) (

*26*,

*27*). Such spin-resolved study is also critical because Hall transport data are now possible that rely heavily on surface state topology and spin configuration data for interpretation (

*15*).

We have carried out spin polarization texture measurements over a large range of binding energies to capture the nonlinear regime that can be accessed by gating for transport measurements. For simplicity, we show results for the hexagonal Fermi surface above the Dirac node and one of the circular Fermi surfaces below the Dirac node of BiTl(S_{0}Se_{1})_{2} (Fig. 2, B and E). Figure 2F shows the measured out-of-plane spin polarization (*Pz*) of cuts C and E (binding energy and momentum direction are defined in Fig. 2, D and E, respectively). No significant out-of-plane spin polarization component is observed within the experimental resolution for cuts such as these. The in-plane measurements, on the other hand, show large polarization amplitudes (Fig. 3A), suggesting that the spin texture is mostly 2D. On the basis of these data, we determine the direction of full 3D spin vectors following a two-step routine (*25*, *28*). Because *P _{z}* = 0 (Fig. 2F), the out-of-plane polar angles are all close to 90°, so only in-plane azimuthal angles are shown in Fig. 3B. On the hexagonal Fermi contour located above the Dirac node (Fig. 3B), the spin vectors obtained from the polarization measurements show that the ground state features a 2D in-plane left-handed chirality spin vortex (a skyrmion in momentum space; Fig. 3, B and C). The direction of the spin is roughly perpendicular to the momentum space track that connects the

*6*) and is also applicable here (

*25*). However, it is not physically possible to realize this chirality inversion or chemical tuning in Bi

_{2}Te

_{3}, because the node is buried under other bands.

The observed chirality inversion of the surface spin texture indicates a 180° turnaround of the spin momentum–locking profile in moving chemical potential across the Dirac node (Fig. 4, A and B). Above the Dirac node, a quasi-particle moving in the +*k* (*k* moving quasi-particle state is locked with *6*, *14*, *29*)—would allow direct observation of the consequences of chirality inversion in the quasi-particle dynamics and interference (Fig. 3).

One particularly interesting proposal involves electrical or chemical gating of both top and bottom surfaces of a topological insulator thin film, resulting in a left-handed chirality (LHC) electron-like Fermi surface on the top and a right-handed chirality (RHC) hole-like Fermi surface of Dirac surface states on the bottom. The interaction between LHC and RHC surface states will lead to excitons at low temperatures with topological properties; the detection of such excitons would allow measurement of the topologically protected fractionalized charge of ±e/2, which can be implemented by optical methods on a device that incorporates the texture inversion (*17*) possible within this materials class (Fig. 4).

To show how these spin textures are relevant in understanding and interpreting quasi-particle transport, such as that observed in the anomalous Hall experiments (*15*) involving the topological insulator surfaces, we explore the key scattering processes on the hexagonal and circular Fermi surfaces observed in our data. Scattering profiles (Fig. 3D) estimated on the basis of the measured spin-ARPES Fermi surfaces and the topological spin textures of BiTl(S_{0}Se_{1})_{2} yield the probability of an electron being scattered in momentum transfer (scattering vector) **q** space (*25*, *30*–*32*). Scattering vector **q** is defined as the momentum transfer from one point on a Fermi surface _{0}Se_{1})_{2} surface consistent with its topological order (*25*). Interestingly, for the hexagonal case, a six-fold symmetric quasi-particle scattering profile is seen, indicating the possibility for a fluctuating spin density wave–type magnetic instability that connects the parallel pieces of the observed spin-polarized Fermi surface (Fig. 3, C and D). This surface magnetic instability is expected to contribute strong fluctuations in the spin transport channel. A recent paper based on mean-field Zener theory showed that a helical magnetic order (HMO) is expected if the surface state topology is in the 3D vectorial spin texture (Fig. 4D) regime; the system can then achieve unusually high critical temperature, even up to *T*_{HMO} ≈ 100 K (*30*). Topological quantization in such a system might then survive at a temperature several orders of magnitude higher than in conventional quantum Hall systems.

## Supporting Online Material

www.sciencemag.org/cgi/content/full/science.1201607/DC1

Materials and Methods

SOM Text

Figs. S1 to S18

## References and Notes

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
See supporting material on
*Science*Online. - ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- We acknowledge discussions with M. Neupane, D. Huse, and D. Haldane. Synchrotron x-ray–based measurements are supported by the Office of Basic Energy Sciences, U.S. Department of Energy (grants DE-FG-02-05ER46200, AC03-76SF00098, and DE-FG02-07ER46352). M.Z.H. acknowledges visiting-scientist support from Lawrence Berkeley National Laboratory and additional support from the A. P. Sloan Foundation. Materials growth and characterization are supported by NSF grant DMR-1006492.