Magnetic Particle Testing (MPT) is a method for determining the presence or absence of a defect by magnetizing the object to be inspected and sprinkling magnetic powder, which is absorbed by the defective part such as a crack and appears as a magnetic powder pattern, which is then evaluated by a specialist. By using the MTP, inspection can be performed without breaking the object to be inspected. However, there are some problems such as the possibility of overlooking defects. In this paper, to solve the problems we develop a classification method of defect images by deep learning for the automation of MPT. The proposed method is based on the structure of U-Net, which has excellent segmentation capability in image processing, and performs segmentation using an improved model that adds convolutional layers to U-Net. Then, an algorithm that combines the result with the last part of the encoder of U-Net is used to discriminate the presence or absence of defects. Using this method, defects were classified from the images obtained during MPT. The results showed that Accuracy of 85.8%, TPR of 65.2%, and FPR of 13.8%.