Absence of magnetic order and emergence of unconventional fluctuations in the Jeff=12 triangular-lattice antiferromagnet YbBO3

We present the ground state properties of a new quantum antiferromagnet, ${\mathrm{YbBO}}_{3}$, in which the isotropic ${\mathrm{Yb}}^{3+}$ triangular layers are separated by a nonmagnetic layer of partially occupied B and O(2) sites. The magnetization and heat capacity data establish a spin-orbit entangled effective spin ${J}_{\mathrm{eff}}=\frac{1}{2}$ state of ${\mathrm{Yb}}^{3+}$ ions at low temperatures, interacting antiferromagnetically with an intralayer coupling $J/{k}_{\mathrm{B}}\ensuremath{\simeq}0.53$ K. The absence of oscillations and a $1/3$ tail in the zero-field muon asymmetries rules out the onset of magnetic long-range order as well as spin freezing down to 20 mK. An anomalous broad maximum in the temperature-dependent heat capacity with an unusually reduced value and a broad anomaly in the zero-field muon depolarization rate centered at ${T}^{*}\ensuremath{\simeq}0.7J/{k}_{\mathrm{B}}$ provide compelling evidence for a wide fluctuating regime $(0.2\ensuremath{\lesssim}T/J\ensuremath{\le}1.5)$ with slow relaxation. We infer that the fluctuating regime is a universal feature of highly frustrated triangular-lattice antiferromagnets while the absence of magnetic long-range order is due to perfect two-dimensionality of the spin lattice protected by nonmagnetic site disorder.