Cooperative Learning Strategy for Human Behavior Prediction Using Multi-Modal Data

Understanding human behaviors leads to fully-automated systems in the near future. This paper investigates a deep learning solution that forecasts human activity patterns based on sensing signals measured by internet-of-things devices. Practical limitations on these small-form-factor sensors request remote deep learning services at a distant edge computing server. Therefore, we need to involve impairments in sensor-server communication phases, such as random packet loss, resource constraint, and propagation noise, in the design of the remote learning architecture. To address these challenges, we propose a collaborative learning strategy among the sensors and server. Each sensor is equipped with its own encoding neural network that compresses high-dimensional sensing signals to communication messages. These are forwarded to the server through imperfect backhaul channels. Then, a classifier at the server infers desired labels. A joint training mechanism of the encoders and classifier is developed along with the channel impairment. By doing so, we can obtain a robust prediction model for arbitrary communication noises. Numerical results demonstrate the viability of the proposed methods.