Fish Disease Cross-Modal Image-Text Retrieval Model in Aquaculture

Highlights A novel RA-FNE model is proposed for recognizing common perch diseases. The PCAA attention mechanism enhances the model’s local lesion recognition. A new RCMM module improves lesion localization and enhances feature representation. The R@1 score in the image-to-text retrieval task reaches 90.7%, improving by 4.9% over the baseline. ABSTRACT. In aquaculture, the timely and accurate recognition of fish disease is essential for early warning and effective prevention. By correlating images with textual semantic descriptions, cross-modal image-text retrieval can significantly improve the efficiency of fish disease recognition. This study mainly focuses on four common diseases of perch: ulcerative disease, saprolegniasis, nocardiosis, and gas bubble disease. On the basis of ViT and BERT, a cross-modal image-text retrieval model named RA-FNE is proposed. Firstly, the partial class activation attention is introduced to enhance lesion localization. Secondly, a rectangular self-calibration mask module is proposed so as to fuse global contextual information from axial pooling and local detailed features extracted by deep convolution. Finally, the convolution and attention fusion module is applied to optimize cross-scale feature interaction and nonlinear representation learning. Experimental results show that the proposed RA-FNE model achieves 90.7% R@1 in the image-to-text retrieval task, with an improvement of 4.9% over the baseline. Meanwhile, it reaches 88.6% R@1 in the text-to-image retrieval task, with an improvement of 3.1%, which can provide reliable technical support for developing intelligent fish disease recognition and prevention systems. Keywords: Cross-modal, Fish disease recognition, Image-text retrieval, RA-FNE.