CRFusion: Fine-Grained Object Identification Using RF-Image Modality Fusion

Object identification is a pivotal enabling technique for smart home and manufacturing applications. Traditional methodologies for object identification predominantly rely on a singular sensor modality, which inherently limits their ability to furnish a detailed characterization of the target object. Addressing this deficiency, in this paper, we fill this gap by introducing CRFUSION, the first-of-its-kind system that integrates the object RGB image and the radio frequency (RF) signal reflected by the object for fine-grained object identification. CRFUSION leverages the complementary characteristics between visible light and radio frequency modalities to simultaneously determine the category and material of target objects. We design a multifaceted object feature from the RF signal, called the Energy Reflection Factor (ERF), which not only reveals the object texture but complements the image modality for identifying the object category. By integrating the characteristics of radar, we obtain radar feature maps based on the ERF of target objects. Additionally, we have developed a modality fusion network to comprehensively integrate the image and ERF features. We conducted a comprehensive evaluation of CRFUSION using a commercial mmWave radar development board and camera. The results show that CRFUSION achieves a classification accuracy of over 96%, demonstrating its robustness, and potential for application.