title: Competing topological and Kondo insulator phases on a honeycomb lattice.
abstract: We investigate the competition between the spin-orbit interaction of itinerant electrons and their Kondo coupling with local moments densely distributed on the honeycomb lattice. We find that the model at half-filling displays a quantum phase transition between topological and Kondo insulators at a nonzero Kondo coupling. In the Kondo-screened case, tuning the electron concentration can lead to a new topological insulator phase. The results suggest that the heavy-fermion phase diagram contains a new regime with a competition among topological, Kondo-coherent and magnetic states, and that the regime may be especially relevant to Kondo lattice systems with 5d-conduction electrons. Finally, we discuss the implications of our results in the context of the recent experiments on SmB(6) implicating the surface states of a topological insulator, as well as the existing experiments on the phase transitions in SmB(6) under pressure and in CeNiSn under chemical pressure.
authors.affiliation: Condensed Matter Group, Department of Physics, Hangzhou Normal University, Hangzhou 310036, China.
authors.forename: Xiao-Yong
authors.initials: XY
authors.lastname: Feng
authors.forename: Jianhui
authors.initials: J
authors.lastname: Dai
authors.forename: Chung-Hou
authors.initials: CH
authors.lastname: Chung
authors.forename: Qimiao
authors.initials: Q
authors.lastname: Si