Towards Pervasive Sensing

Channel state information (CSI) and RGB (Red, Green, Blue) image are two fundamental modalities for accurate human activity recognition (HAR). CSI data is privacy preserving, independent of viewing angle and lighting, while RGB image data can provide useful information after processed by deep learning (DL) methods. However, both modalities have inherent disadvantages. CSI modality is with high environmental dependence, causing unstable fluctuations in wireless signals. RGB image modality is heavily restricted by camera facing angle and lighting condition. Hence, we propose a general multimodal fusion framework for HAR tasks based on both CSI and RGB image modalities. Distinct from other literatures on multimodal fusion mainly based on vision modalities, we utilized CSI modality from wireless domain along with RGB images to realize more comprehensive human activity sensing. Both early fusion and late fusion strategies are formulated and developed, with multiple models constructed from two modern unimodal frameworks to constitute corresponding fusion models. Experiments are conducted on a synchronous CSI and RGB image human activity dataset and validate the advantage of the proposed multimodal fusion design comparing with unimodal approaches.