To address the optimization problem of complex gear parameters, this paper proposes a novel genetic algorithm-enhanced efficient global optimization method. Firstly, obtain initial sampling points through latin hypercube sampling (LHS) and construct a Kriging surrogate model. Subsequently, a genetic algorithm (GA) is employed for global exploration to identify the sampling point with the maximum expected improvement (EI) value. On this basis, local exploitation is used to refine the search results. The EI value of the new sampling point is evaluated and incorporated into the sample set, updating the surrogate model to enhance prediction accuracy. Finally, this process is repeated until convergence criteria are met, yielding the optimized results. In theoretical case studies, optimization of the Hartman3 and Hartman6 functions demonstrates that the proposed algorithm outperforms other surrogate-based optimization methods, such as EGO, GA-ANN, and GA-RSM, in terms of global optimization efficiency and accuracy, while exhibiting high robustness across multiple runs. Furthermore, leave-one-out cross-validation (LOOCV) confirms the high accuracy of the Kriging model’s predictions compared to true values in the global optimum region. When applied to the optimization of a planetary gear train, the gear train volume is reduced by 15.1%, and the failure probability is also decreased, highlighting the method’s ability to balance system lightweighting and reliability. This approach provides an efficient solution for the design of high-performance planetary gear transmission systems, expanding the application potential of surrogate models and global optimization algorithms in mechanical design.