Classifying Major Depressive Disorder and Response to Deep Brain Stimulation Over Time by Analyzing Facial Expressions

Objective: Major depressive disorder (MDD) is a common psychiatric disorder that leads to persistent changes in mood and interest among other signs and symptoms. We hypothesized that convolutional neural network (CNN) based automated facial expression recognition, pre-trained on an enormous auxiliary public dataset, could provide improve generalizable approach to MDD automatic assessment from videos, and classify remission or response to treatment. Methods: We evaluated a novel deep neural network framework on 365 video interviews (88 hours) from a cohort of 12 depressed patients before and after deep brain stimulation (DBS) treatment. Seven basic emotions were extracted with a Regional CNN detector and an Imagenet pre-trained CNN, both of which were trained on large-scale public datasets (comprising over a million images). Facial action units were also extracted with the Openface toolbox. Statistics of the temporal evolution of these image features over each recording were extracted and used to classify MDD remission and response to DBS treatment. Results: An Area Under the Curve of 0.72 was achieved using leave-one-subject-out cross-validation for remission classification and 0.75 for response to treatment. Conclusion: This work demonstrates the potential for the classification of MDD remission and response to DBS treatment from passively acquired video captured during unstructured, unscripted psychiatric interviews. Significance: This novel MDD evaluation could be used to augment current psychiatric evaluations and allow automatic, low-cost, frequent use when an expert isn't readily available or the patient is unwilling or unable to engage. Potentially, the framework may also be applied to other psychiatric disorders.