Modal Deep Learning for Tongue Diagnosis

Tongue diagnosis is a key component of traditional medical systems and increasingly relevant in intelligent healthcare. This study proposes a multi-modal approach integrating visual and textual data to classify tongue images into three diagnostic categories: Normal, HF Tip, and Abnormal. We compare the performance of three deep learning architectures: Convolutional Neural Network (CNN) for image-based classification, Long Short-Term Memory (LSTM) for text-only processing, and a hybrid CNN+LSTM model for multi-modal fusion. A 10-fold cross-validation was conducted on a curated dataset of 1,273 tongue images and corresponding medical descriptions. Results demonstrate that the fusion model significantly outperforms unimodal models with an accuracy of 98.11%, indicating the effectiveness of combining image and language features in automated diagnosis systems.