The rotor system of large-scale rotary machine is an important constituent part of the whole unit.Based on the analysis for the rotor,bearings,supporting system vibration,it sets up the vibration model of the rotor system and the mechanical model of lumped mass rotor system.Meanwhile optimization design for the rotor system is made with multi-mode vibration as the goal,the multi-critical speed and dynamic response as the bounded variable.And then the design method is optimized from mathematical modeling to optimization analysis,and systematic expatiation is made by using ANSYS calculations.Through the design,the dynamic performance of the rotor system is predicted and optimized in drawings design stage so that performance tends to be optimal.