The enhancement of surface quality in ultra-precision machining of aluminum alloys is frequently impeded by coarse grains and second-phase particles. This study investigated microstructural refinement of 6061 aluminum alloy through cryogenic multi-directional forging combined with heat treatment to enhance surface quality in single point diamond turning (SPDT). Samples underwent initial annealing (AT) or solution treatment (ST), followed by 1–9 passes of cryogenic forging, solution treatment, and artificial aging. Microstructural characterization revealed that, compared to AT pretreatment, ST pretreatment suppressed dynamic recovery under cryogenic deformation, yielding higher dislocation density and substructure formation. These provided additional nucleation sites for static recrystallization in subsequent solution treatment, resulting in finer, equiaxed grains. Cryogenic forging fragmented coarse second-phase particles, while solution treatment enhanced their dissolution, reducing particle size and volume fraction. Increased dissolution into the aluminum matrix promoted a more uniform precipitation state, which contributed to improved hardness uniformity. SPDT experiments demonstrated that the refined microstructure reduced grain-boundary steps and defects associated with second-phase particles. Compared with the AT1 sample, the ST9 sample exhibited a decrease in the arithmetic average surface roughness, Ra, from 12.54 nm to 6.81 nm, corresponding to a reduction of 45.7%.