We report deep learning-based design of diffractive neural networks. Following its fabrication, a diffractive neural network can all-optically perform user-defined tasks through diffraction. We demonstrate the applications of this framework for classification and imaging tasks.