Passive acoustic monitoring is an effective method for observing biological and environmental activity in the ocean. Many of vocalization characteristics of aquatic mammals have been reported by far. However, manual classification of biological sounds is labor-intensive, making automated methods with strong generalization essential. We developed a supervised machine learning model to classify whistles of striped dolphin and other underwater sounds recorded in Suruga Bay, Japan. We also evaluated model explainability using Explainable Artificial Intelligence to assess feasibility of automated classification in practical settings. With a confidence threshold of 0.95 for detecting the whistles, the model achieved precision of 0.961 and recall of 0.928. Explainability analyses showed that the model sometimes focused on non-whistle features, indicating limited generalization. To improve this, we applied spectral gating to reduce background noise and retrained the model. The updated model improved performance, achieving precision of 0.967 and recall of 0.940, and showed reduced focus on irrelevant features. This study demonstrates the potential of explainable machine learning for underwater bioacoustic classification. As future work, we aim to expand the system to multiple species and noise environments and to explore zero-shot and few-shot learning for greater adaptability.