Abstract Fault mechanism feature analysis of planetary gear trains using vibration signals is of great significance. However, affected by numerous internal excitation forces and complex external transfer paths, the corresponding vibration features of planetary gear trains are weakened and become much more complicated, especially existing crack faults in gears. Aiming at resolving this problem, a rigid–flexible coupling dynamics model and a signal convolution model of planetary gear trains are proposed to reveal the vibration modulation mechanisms under a cracked state. Firstly, numerous internal excitation forces that are mainly derived from the meshing stiffness are calculated through a combination of the lumped parameter model and energy method. Secondly, multiple external transfer paths with full consideration of flexible housing are acquired by the finite-element-method, whose time-varying properties are described by modal parameters with clear physical meaning. Finally, based on the constructed signal convolution model, the full-band vibration signals with normal and cracked planet gears are synthesized through the convolution of multiple excitation and multiple paths. We conclude that three groups of resonance modulation features effectively indicate the planet crack state in simulations, and the corresponding vibration response mechanism is further revealed. The experimental signals of planetary gear trains also verify the correctness of the proposed models and the concluded modulation features.