Data augmentation in material images using the improved HP-VAE-GAN

Since the rapid development of computer vision relies heavily on large-scale labeled data and high-performance computing equipment, therefore, image recognition in small sample datasets faces several challenges, such as difficult to implement model training. In the field of materials research, the cost of collecting image data is relatively high. In order to solve the problem of insufficient image samples in material research, an improved HP-VAE-GAN is proposed to generate material images to achieve data augmentation. HP-VAE-GAN is a single sample generation model that consists of Patch-VAE and Patch-GAN. The improved HP-VAE-GAN introduces the attention mechanism into model. By adding CBAM (Convolutional Block Attention Module) to the encoder of Patch-VAE, the feature extraction and representation capabilities of the network are further improved. Use this model to train a single image, and then generate a certain number of samples to achieve the expansion of the training set. For the classification of ultrahigh carbon steel microstructure images, experiments show that the accuracy of classification model (MobileNet, ResNet50 and VGG16) trained with real images plus generated images is improved obviously. In addition, the effectiveness of the improved HP-VAE-GAN is verified by experiments on texture images similar to material images.