Microorganisms play a key role in fish spoilage and quality deterioration, making the development of a rapid, accurate, and efficient technique for detecting surface microbes essential for enhancing freshness and ensuring the safety of mandarin fish consumption. This study focused on the total viable count (TVC) and Escherichia coli levels in the dorsal and ventral parts of fish, and we constructed a detection model using hyperspectral imaging and data fusion. The results showed that comprehensive and simplified models were successfully developed for quantitative detection across all wavelengths. The models performed best at predicting microbial growth on the dorsal side, with the RAW-CARS-PLSR model proving the most effective at predicting TVC and E. coli counts in that region. The RAW-PLSR model was identified as the optimal predictor of the E. coli concentration on the ventral side. A fusion model in the decision layer constructed using the Dempster–Shafer theory of evidence outperformed models relying solely on spectral or textural information, making it an optimal approach for detecting surface microbes in mandarin fish. The best prediction accuracy for dorsal TVC concentration achieved an Rp value of 0.9337, whereas that for ventral TVC concentration reached 0.8443. For the E. coli concentration, the optimal Rp values were 0.8180 for the dorsal section and 0.8512 for separate analysis.