A Multimodal Multi-Drone Cooperation System for Real-Time Human Searching

Aerial images from drones have been used to search individuals in the crowd. However, using a single drone for human searching faces challenges including low accuracy and long latency, due to poor visibility and limited on-board computing resources. In this paper, we propose SkyNet, a multi-drone cooperation system for real-time human searching, including locating and identifying. To locate a person, SkyNet uses Multi-View Cross Search with only 2D images. To achieve accurate identification, SkyNet processes faces in images from multi-view drones in three steps. First, a Multi-Modal Face Correction is designed to transform less useful face into desired target face, guided by text instructions. Second, an Angle Masking Network is developed to minimize invalid data of a single profile face. Third, the multiple face from drones are fused by a Fusion Weight Network. Moreover, by predicting the estimated finishing time of tasks, SkyNet schedules and balances workloads among edge devices and the cloud server to minimize processing latency. We implement SkyNet in real life, and evaluate the performance with 20 human participants. The results show that SkyNet can locate people within 0.18m error. The identification accuracy reaches 95.87%, and the system process is completed within 0.84s.