Topological phase transition in Sb-doped Mg3Bi2 monocrystalline thin films

${\mathrm{Mg}}_{3}{\mathrm{Bi}}_{2}$ has been proved to be a semimetal with topological surface states (referred to as a topological semimetal, TSM) in the presence of spin-orbit coupling (SOC), and predicted to be a type-II nodal line semimetal (NSM) in the absence of SOC. It is possible to tune the effective SOC in ${\mathrm{Mg}}_{3}{\mathrm{Bi}}_{2}$ by substituting the heavy Bi atom with lighter atoms, such as Sb or As, which results in topological phase transitions. Here in our work we investigate the evolution of the transport properties and band structures of ${\mathrm{Mg}}_{3}{\mathrm{Bi}}_{2}$ by doping different concentrations of Sb through molecular beam epitaxy. Our results demonstrate the existence of phase transitions of TSM to trivial semimetal (SM) and SM to trivial insulator, with Sb concentration $x$ at ${x}_{1}\ensuremath{\approx}0.44--0.64$ and ${x}_{2}\ensuremath{\approx}1.39$, respectively. Further theoretical calculations give values of ${x}_{1}\ensuremath{\approx}0.523$ and ${x}_{2}\ensuremath{\approx}1.10$, which agrees well with our experimental results. And all these results are concluded into a ${\text{Mg}}_{3}{\text{Bi}}_{2\ensuremath{-}x}{\text{Sb}}_{x}$ phase diagram. Our work will stimulate more explorations to the possibilities of the realization of type-II NSM in ${\mathrm{Mg}}_{3}{\mathrm{Bi}}_{2}$ by various elements substitution.