Multimodal Fusion for Precision Personality Trait Analysis: A Comprehensive Model Integrating Video, Audio, and Text Inputs

The multi-modal personality trait analysis system aims to look at the association between personality characteristics, speech, body language, and facial expressions. Subsequently, many techniques for gathering data have been employed, such as capturing individuals in various social contexts, doing interviews, and carrying out personality tests. The data has been meticulously labelled to enable meaningful analysis of the modalities. In this study, we took a holistic approach, combining BERT for text classification, Mel-frequency cepstral coefficients with CNN for audio analysis, and Convolutional Neural Networks for image processing. In our examination, this integrated approach demonstrates the sophisticated use of natural language comprehension, audio signal processing, and computer vision. Machine learning methods, such as deep neural networks and clustering approaches, have been used in personality trait analysis to find patterns and correlations between the modalities and personality traits. The accuracies for individual modalities in analysing a given video are image - 0.689654, audio - 0.665432, and text - 0.801234. Following a late fusion process that combines data from these modalities, the classification of content based on the “big five” personality traits achieves an overall accuracy of 0.802345.