Voice Phishing (vishing) Detection System

With the rapid advancement of AI-powered communication tools, voice phishing — commonly known as vishing — has emerged as one of the most deceptive forms of cybercrime. Attackers impersonate trusted entities over phone calls to manipulate victims into disclosing sensitive data such as banking credentials, OTPs, and personal identification details. Conventional defenses like caller ID screening and spam number databases are no longer adequate, as they fail to analyze the actual content and emotional tone of a conversation. This paper presents a multimodal AI system that addresses this gap by examining both the acoustic properties and linguistic content of a phone call. Audio input is first transcribed using Whisper ASR, after which TF-IDF-based Naive Bayes classification is applied on the transcript for textlevel phishing detection. Simultaneously, MFCC features extracted from the raw audio are fed into a Random Forest classifier to detect emotional cues such as urgency and stress. A fusion decision engine consolidates both outputs to produce a final verdict — Vishing or Safe — along with a confidence score. The system is deployed as a Streamlit web application. Experimental results show spam detection accuracy of 98.7%, emotion detection accuracy of 69.8%, and an overall system accuracy in the range of 90–95%.