The emergence of van der Waals (vdW) magnets has opened new frontiers for investigation of 2-dimensional magnetism and application of spintronic memories. However, as the system approached the nanoscale, the edge effect significantly enhanced, posing critical challenges in understanding edge-mediated modifications of magnetic behavior, such as the nanoscale magnetic domain. Here, we employ magnetic force microscopy to systematically investigate edge-mediated magnetic domain rearrangement in Fe3GaTe2 flakes. By performing thermal treatment, we observed a distinct transition from labyrinthine to edge-localized stripe domains perpendicular to the flake edges. Furthermore, we reveal a competition and stabilization between size effect and edge effect by investigating thickness-dependent variation in the length of stripe domains and indicate that such an edge effect suppresses the formation of skyrmions. These findings not only elucidate the critical role of the edge effect in governing the domain morphology of vdW magnets but also provide insights for designing edge-mediated nanostructures in next-generation spintronic devices.