End-to-end cross-species whistle detection via transformer-based instance segmentation

Detection and fine-scale annotation of dolphin whistles holds significant biological and ecological importance for understanding marine mammal communication patterns and population dynamics. Dolphin whistle detection faces challenges from discontinuous, faint signals in noisy backgrounds. Most existing annotation methods treat the task as a spectrogram peak tracking problem, constructing whistle trajectories from neighboring detected peaks by using heuristic rule-based methods or statistical models, and rely on manual feature engineering. While effective for long, clear whistles, they lack generalizability and do not perform as well on short, weak whistles across diverse species. We reformulate dolphin whistle detection as an instance segmentation problem and propose a transformer-based model to directly predict individual whistles, eliminating the need for separate peak detection and trajectory formation steps. To address limitations of annotations in existing datasets, we also implement a human-in-the-loop training paradigm that iteratively refines data labeling, improving both dataset quality and model performance throughout the training process. Our end-to-end model demonstrates robust detection capabilities that generalize effectively across whistle lengths and multiple dolphin species. Experimental results show significant advancement: 92.20% precision and 73.53% recall on all whistle lengths, and 88.92% precision with 85.07% recall on whistles longer than 150 ms and SNR above 10 dB.