mmFall: Fall Detection Using 4-D mmWave Radar and a Hybrid Variational RNN AutoEncoder

Elderly fall prevention and detection becomes extremely crucial with the fast aging population globally. In this article, we propose mmFall, a novel fall detection system, which comprises 1) the emerging millimeter-wave (mmWave) radar sensor to collect the human body’s point cloud along with the body centroid and 2) a hybrid variational recurrent neural network (RNN) autoencoder (HVRAE) to compute the anomaly level of the body motion based on the acquired point cloud. A fall is detected when the spike in anomaly level and the drop in centroid height occur simultaneously. The mmWave radar sensor offers privacy-compliance and high sensitivity to motion, over the traditional sensing modalities. However, 1) randomness in radar point cloud and 2) difficulties in fall collection/labeling in the traditional supervised fall detection approaches are the two major challenges. To overcome the randomness in radar data, the proposed HVRAE uses variational inference, a generative approach rather than a discriminative approach, to infer the posterior probability of the body’s latent motion state every frame, followed by a RNN to summarize the temporal features over multiple frames. Moreover, to circumvent the difficulties in fall data collection/labeling, the HVRAE is built upon an autoencoder architecture in a semisupervised approach, which is only trained on the normal activities of daily living (ADL). In the inference stage, the HVRAE will generate a spike in the anomaly level once an abnormal motion, such as fall, occurs. During the experiment,¹ we implemented the HVRAE along with two other baselines, and tested on the data set collected in an apartment. The receiver operating characteristic (ROC) curve indicates that our proposed model outperforms baselines and achieves 98% detection out of 50 falls at the expense of just 2 false alarms. Note to Practitioners—Traditional nonwearable fall detection approaches typically make use of a vision-based sensor, such as camera, to monitor and detect fall using a classifier that is trained in a supervised fashion on the collected fall and nonfall data. However, several problems render these methods impractical. First, camera-based monitoring may trigger privacy concerns. Second, fall data collection using human subjects is difficult and costly, not to mention the impossible ask of the elderly repeating simulated falls for data collection. In this article, we propose a new fall detection approach to overcome these problems 1) using a palm-size mmWave radar sensor to monitor the elderly, that is highly sensitive to motion while protecting privacy and 2) using a semisupervised anomaly detection approach to circumvent the fall data collection. Further hardware engineering and more training data from people with different body figures could make the proposed fall detection solution even more practical.