Multi-scale GAN with transformer for surface defect inspection of IC metal packages

Integrated circuit plays an important role in the information technology industry. Surface defect inspection of IC packages is an essential process in the IC packaging manufacturing. Here, an automatic optical inspection system is proposed based on semi-supervised deep learning for surface defect inspection of IC metal packages. Different from previous inspection methods, we propose an entirely multi-scale inspection framework to implement the evaluation of defects at multiple scales. To well capture the intrinsic patterns of qualified samples at multiple scales, an entirely multi-scale GAN with transformer is elaborately designed, which incorporates several novel modules. Specifically, multi-scale CNN encoder with a novel feature extraction scheme and a cross-scale feature fusion module is designed to sufficiently extract the features from the IC metal package image. Different from previous GAN models, a Swin Transformer decoder is designed to strengthen the modeling ability of the proposed GAN model. Also, several novel multi-scale inspection schemes, including multi-scale weight mask, multi-scale adaptive thresholding and multi-scale image patch-based defect evaluation, are proposed to suppress the reconstruction errors, highlight the potential defects and further evaluate them at multiple scales, respectively. Experimental results demonstrate the effectiveness and feasibility of the proposed multi-scale inspection framework, which achieves an excellent inspection performance of 0.70% error rate, 0.57% omission rate, 99.3% accuracy, 99.8% precision, 99.3% recall and 0.996 F-score with a reasonable speed of 70.9 FPS and is superior to the state-of-the-art semi-supervised deep learning inspection methods.