Lesion identification in fundus images via convolutional neural network-vision transformer

Classification of fundus lesions plays a vital role in detecting some diseases in their early stages, including glaucoma and diabetes. During the clinical diagnosis and treatment prognosis, its higher accuracy will assist the doctors in optimizing the therapeutic schedule and reduce the workload of the ophthalmologists. However, the intrinsic characteristics of retinal lesions in retinal images make the detection process challenging. In recent decades, many deep learning algorithms have been widely applied in various areas and achieved promising outcomes. Among the deep learning approaches, due to the overall success of Transformers in the natural language processing field, plenty of researchers have begun to explore the applicability of Transformer models in clinical applications such as recognizing various ophthalmic diseases. In this study, we propose a vision transformer-based pipeline for accurately classifying retinal diseases. To fully exploit the local and global associations between the individual image patches, a convolutional neural network concatenated with a vision transformer is employed to form the proposed framework. It can improve the performance of retinal lesion classification compared with a single-vision transformer or a convolutional neural network. Moreover, we pre-train our proposed framework on the ImageNet ISLVRC dataset and a sizeable retinal image database in sequence. Then, we fine-tune this vision transformer on downstream fundus image classification tasks. Experimental results demonstrate that the proposed approach achieves superior performance on two publicly available datasets over state-of-the-art deep learning-based techniques, including convolutional neural networks and other vision transformers.