WiVi-GR: Wireless-Visual Joint Representation Based Accurate Gesture Recognition

Human gesture recognition provides great potentials in Human Computer Interaction (HCI), and the wireless or visual signals based technologies have been explored in their respective fields. The intrinsic characteristics of both modalities are complementary to each other, e.g. the wireless signal is robust to illumination changes and occluded conditions but suffers from low space resolution, while the visual signal has high space resolution but vulnerable to scenario variations. Intuitively, integrating the two modalities has potential chance to improve the overall discriminative power. However, existing multi-modal fusion methods could not fully exploit their complementarity to achieve accurate estimation, and also lack physical interpretability. In order to solve this issue, we introduce WiVi-GR: a Wireless-Visual joint representation based accurate Gesture Recognition system, which constructs a complete velocity representation to guarantee robust and accurate gesture recognition. Specifically, we analyze the complementarity of the two modalities in data dimension and spatial-temporal feature resolution, and propose an Interpretable Orthogonal Representation (IOR), which applies multi-channel coding to get image plane velocity, utilizes frequency domain analysis to get radial velocity, and aggregates both to achieve the complete representation of the dynamic pattern. Based on the IOR, we perform a data-level fusion with channel superposition convolutions to accomplish the accurate gesture recognition task. Experimental results show that the proposed WiVi-GR outperforms traditional multi-modal approaches by large margins, especially in small training sample set condition.