Deep neural network with adaptive dual-modality fusion for temporal aggressive behavior detection of group-housed pigs

• Temporal aggressive behavior detection for monitoring behavior of group-housed pigs . • Proposed method can detect occurring and temporal interval of aggressive behaviors. • Adaptive dual-modality fusion module assigned weights to spatial and motion feature. The frequent occurrence of pig aggressive behaviors in intensive group-housed environment seriously affects pig health, welfare and farms economy. Accurate detection of the occurring and temporal interval of aggressive behaviors is important for pig farming. The study aimed to develop an automatic temporal aggressive behavior detection method based on deep neural network. This network mainly consists of three modules, i.e., aggression feature extraction, adaptive dual-modality fusion and aggression temporal proposal generation. First, RGB data and optical flow data was used to extract the spatial and motion information of pig aggressive behaviors. Second, a modality attention and a temporal attention were specifically designed to adaptively fuse features of different modalities. Third, an anchor-free aggression temporal proposal generation strategy was applied to generate aggression proposals, which indicate the start and end times of aggressive behavior. To evaluate the proposed method, a behavior dataset containing 216 videos and 642 annotations was constructed. On the test set, this method achieves an AP value of 68.0 %, an AR value of 77.8 % in average number of proposals at 100. To test this method in practical application, our method was conducted on an additional 90 min untrimmed surveillance video and effectively predicted the real aggression instances. The results demonstrated that it can meet the practical needs of intelligent monitoring in pig farming and analysis in animal behavior research. We shared our temporal aggressive behavior detection dataset at https://github.com/IPCLab-NEAU/Temporal-Aggressive-Behavior-Detection for precision livestock farming research community.