Abstract Gear transmission plays a pivotal role in determining the efficiency and reliability of mechanical systems. As an emerging mathematical framework, fractal theory provides powerful tools for describing and analyzing the complex, multiscale characteristics of gear surfaces and their interactions. However, a systematic and critical synthesis of its applications in gear transmission remains lacking. This review comprehensively summarizes recent advances in fractal-based research, encompassing tooth surface characterization, contact mechanics, friction and wear, dynamic modeling, and fault diagnosis. The limitations of existing studies are discussed, such as the scarcity of experimental validation, insufficient exploration of fractal-elastohydrodynamic lubrication (EHL) coupling. Furthermore, potential research directions are proposed, including dynamic identification and real-time estimation of fractal parameters, as well as the integration of fractal features with deep learning models. These developments are expected to enhance the accuracy and reliability of gear transmission system modeling and broaden the practical applicability of fractal theory in mechanical engineering.