Quantum anomalous Hall effect and topological phase transition in two-dimensional antiferromagnetic Chern insulator NiOsCl6

By doing calculations based on density functional theory, we predict that the two-dimensional anti-ferromagnetic (AFM) NiOsCl6 as a Chern insulator can realize the quantum anomalous Hall (QAH) effect. We investigate the magnetocrystalline anisotropy energies in different magnetic configurations and the Néel AFM configuration is proved to be ground state. When considering spin-orbit coupling (SOC), this layered material with spins perpendicular to the plane shows properties as a Chern insulator characterized by an inversion band structure and a nonzero Chern number. The nontrivial band gap is 37 meV and the Chern number C = -1, which are induced by a strong SOC and AFM order. With strong SOC, the NiOsCl6 system performs a continuous topological phase transition from the Chern insulator to the trivial insulator upon the increasing Coulomb repulsion U. The critical U c is indicated as 0.23 eV, at which the system is in a metallic phase with [Formula: see text]. Upon increasing U, the E g reduces linearly with C = -1 for 0 < U < U c and increases linearly with C = 0 for U > U c . At last we analysis the QAH properties and this continuous topological phase transition theoretically in a two-band [Formula: see text] model. This AFM Chern insulator NiOsCl6 proposes not only a promising way to realize the QAH effect, but also a new material to study the continuous topological phase transition.