Well-dispersed Fe3O4 nanoparticles were prepared and modified by oleic acid at the temperature of 70°C for different reaction times. The surface modification of oleic acid could improve dispersion stability of Fe3O4 nanoparticles in vegetable oil by forming an effective chemisorbed modification layer on the surface of nanoparticles. The interaction between coordination pattern and mass faction of chemisorbed oleic acid on nanoparticles and dispersion stability of nanofluid was discussed. The optimal dispersion stability of vegetable-oil based Fe3O4 nanofluids was obtained to the nanoparticles modified for 8 hours, which has the highest AC breakdown strength of 67.78 kV.