Fine-Grained Question-Level Deception Detection via Graph-Based Learning and Cross-Modal Fusion

Automated deception detection has been found as a vital and concerned task, capable of assisting human users to assess truthfulness and detect deceptive behaviors in several situations (e.g., medical, legal, as well as occupational domains). As contact-free video cameras and data analysis techniques are leaping forward, leveraging one's visual, acoustic and textual information captured in a video can be cost-effective for deception detection. In this study, we aim at a fine-grained question-level deception detection task, focusing on detecting whether a subject lies or not in answering each question rather than providing an overall "lie or not" judgement. A Graph-based Cross-modal Fusion Model (GCFM) is presented to learn the inherent associations among the subject's reactions to different questions, plus a novel cross-modal attention mechanism to enhance the model's learning capability. As revealed by the experimental results on the two datasets, the proposed graph-based GCFM outperformed eight other methods, and its two alternatives (clustering K-means based and sequential learning LSTM based methods), achieving accuracy 88.14% and 86.91% on the two datasets, respectively. Besides, through association learning, GCFM could increase the accuracy by 1.87% and 4.33% on the two datasets, respectively. Furthermore, its cross-modal attention mechanism led to the improvement of accuracy by 2.44% and 2.95% on the two datasets, respectively.