The transverse vibration generated during the operation of the ship shaft interferes with the operation of the equipment and accelerates the aging of bearing components. It also intensifies shaft-shell vibration and increases the level of underwater radiated noise of the ship. Aiming to solve the problem of controlling the transverse vibration of the ship propulsion shaft, an active control method for suppressing the transverse vibration of the shaft is proposed by balancing the control error and convergence speed of the FxLMS algorithm. The method’s effectiveness is demonstrated through numerical simulations of active vibration control and shaft bench tests. The experimental results show that the control using the improved algorithm can improve the noise reduction effect of each main characteristic spectral line by more than 4 dB.