Developing an Artificial Intelligence Chatbot for Snake Image Classification and Accuracy Improvement

ABSTRACT. Snakebites pose a significant global health challenge. The timely and accurate identification of snake species is essential for guiding antivenom administration. Our goal was to evaluate the effectiveness of a machine learning model for classifying snake species using images collected from the external environment and a preprocessing method to enhance accuracy. In this study, we developed a deep learning model for snake species identification in Taiwan based on the Swin Transformer v2 architecture, applying transfer learning to 12,000 images sampled from a dataset of 30,573 labeled images collected by the authors from sources such as Flickr, iNaturalist, and local databases before October 2023. An external test set of 2,400 images, collected through the LINE chatbot and Facebook groups between November 2023 and April 2024, was used to evaluate real-world performance. To address challenges in external test set images, we introduced a preprocessing method called test-time object detection and cropping (TT-ODC). Without preprocessing, the model achieved 95.6% accuracy on the validation set but dropped to 83.3% on the external test set. Applying TT-ODC improved external test accuracy to 89.8%, closely matching human annotation performance (90.3%). These findings revealed that integrating a Swin Transformer v2-based model into the LINE chatbot enhances snake species identification and improves real-world accuracy. The TT-ODC method effectively bridges the gap between experimental (validation set) and real-world (external test set) performance, providing a practical tool for clinical snakebite management.