All-optical, helicity-independent magnetization switching (AO-HIS) is promising for future ultrafast, energy-efficient magnetic data storage. Achieving high bit density requires the reduction of optically addressed magnetic bit sizes at deterministic locations. Metallic nanostructures that support localized surface plasmons enable electromagnetic confinement below the diffraction limit. Rare-earth transition metal alloys like GdTbCo support stable nanometer-sized magnetic domains. We fabricate plasmonic gold nanostructures on a GdTbCo film and demonstrate in situ deterministic toggling of magnetic states through optical excitation using magnetic force microscopy. Imaging the magnetic state with nanometer resolution allows us to observe AO-HIS with a minimum width of 240 nm localized at the edges of plasmonic nanobars following excitation by a single ultrashort laser pulse at 1030 nm. The strong optical field localization via plasmonic nanobars enables reproducible AO-HIS on subwavelength scales. Additionally, high-resolution magnetic imaging highlights differences between the magnetic switching pattern for on- and off-resonant excitation of plasmonic nanodiscs.