Spin Frustration and Unconventional Surface Spin Canting State in Van der Waals Ferromagnet/Antiferromagnet Heterostructures

Abstract Atomically flat surfaces of van der Waals (vdW) materials pave an avenue for addressing a long‐standing fundamental issue of how a compensated antiferromagnet (AFM) surface frustrates a ferromagnetic (FM) overlayer in FM/AFM heterostructures. We investigate Fe 5 GeTe 2 /NiPS 3 vdW heterostructures by characterizing AFM and FM spins separately. We find that in‐plane zig‐zag AFM NiPS 3 develops three equivalent AFM domains, which are robust against external magnetic field and magnetic coupling with Fe 5 GeTe 2 . Moreover, evidence is provided of in‐plane‐AFM‐induced perpendicular magnetic anisotropy (PMA) in adjacent Fe 5 GeTe 2 , and an unconventional out‐of‐plane surface spin canting state with the Fe 5 GeTe 2 spins spatially turn from out‐of‐plane direction near the interface to in‐plane direction away from the interface in Fe 5 GeTe 2 /NiPS 3 . The out‐of‐plane surface spin canting is a unique property of spin frustration in vdW magnetic heterostructures.