In computational imaging, existing diffractive imaging system design methods have limitations in efficiently simulating diffraction effects. Moreover, height errors in the lithographic process can cause deviations between the actual height profile and the simulated continuous height profiles, negatively impacting the imaging system's performance. Here, we propose an end-to-end diffraction imaging system design method that accounts for lithographic process characteristics. Specifically, a hierarchical discrete model is used for simulation, constraining the height distribution to positions favorable for lithography, thereby mitigating the negative impact of a height error. Then, a dynamic zero padding method is used in point spread function (PSF) calculation to reduce the redundancy of zero padding points, resulting in a 51.78% increase in speed. Guided by the gradient information from the loss function, the diffractive optical element (DOE) with a 4 mm aperture and the image reconstruction network, i.e., gated Wiener–Lucy chain, are jointly optimized in an end-to-end manner. Simulation experiments show that while the PSNR of the reconstructed images using our learned hierarchical discrete height model is 0.3 dB lower than that of the continuous DOE designed without considering the lithographic process characteristics, it demonstrates greater robustness against height errors. This approach provides a reference for future DOE design methods that integrate different manufacturing process characteristics.