The optical diffractive neural network (ODNN), based on the free-space propagation of light waves, exhibits significant ad-vantages, including ultra-high speed, low power consumption, and parallel computation. However, this technology faces challenges in practical applications, particularly concerning fabrication and alignment accuracy, with stringent requirements on manufacturing processes. In this paper, a class of hybrid optical diffractive neural networks (H-ODNNs) is designed by constructing continuous passive phase modulation layers using diffraction neurons of varying sizes. Three representative tasks (digit recognition, image processing, and wavelength multiplexing) substantiate its superior performance in enhancing robustness. Compared with network with vaccination (a common method for enhancing robustness), the H-ODNN does not need vaccination training, the average training time is reduced by approximately 50%, and even achieves superior performance. Additionally, the larger size of some diffraction neurons, the H-ODNN reduces the complexity of fabrication and improves manufacturing yield. This work provides a new concept for the design of ODNN.