This article proposes a new type of herringbone planetary gear transmission based on gear geometry theory. Considering the special circular-arc and parabolic curve characteristics, the new herringbone planetary gear drive consisting of the circular-arc convex sun gear, parabolic concave planetary gear and circular-arc convex internal gear is proposed to overcome the vibration limitations of traditional involute profiles. Through rigorous kinematic modeling, the original tooth surface equations are derived and a novel dynamic analysis framework for investigating nonlinear characteristics of the proposed planetary gear transmission is provided. Gear stiffness excitation model based on energy method is established. Time-varying mesh stiffness of the new herringbone planetary gear system under single and double teeth conditions is analyzed according to concentration parameter theory. Influences of tooth flank clearance and meshing error on the design results are introduced. Furthermore, dynamic models of the new herringbone planetary gear system are established, and the equations of motion with time-varying coefficients are derived. This study employs a fourth-order variable-step adaptive Runge–Kutta numerical integration algorithm (absolute error tolerance: 1e-6) to solve the governing equations. The dynamic response results reveal that the novel herringbone planetary gear exhibits a maximum sun gear amplitude of 7.31×10−6 mm at 2000 r/min, showing 34.7% vibration reduction in the test conditions, with an 8% average dynamic load coefficient reduction observed across different speeds. The results suggest the vibration spectrum reconstruction capability of point-contact tooth profiles. Comparative analysis with traditional involute herringbone gears shows potential advantages in energy concentration characteristics and sun gear dynamic stability.