Dual-Branch Spatiotemporal Interaction Network for Video Crowd Counting

By utilizing the spatiotemporal correlations of the consecutive frames, video crowd counting methods are usually more accurate and robust than image-based methods. However, they still suffer from two major problems. First, the crowd features are mostly extracted in a single and fixed temporal scale. Second, they often learn the spatial and spatiotemporal features in a serial approach and the interactions between them are mostly neglected. To address the above problems, we present a Dual-branch SpatioTemporal Interaction network (DSTI) for video crowd counting. Specifically, a purely video-based backbone called Meanformer is designed elaborately to establish a long-term, multi-scale and global temporal representation by combining the strengths of 3D convolution and transformer. Considering the lack of pre-trained weights, a spatiotemporal full-connected composition operation is proposed to boost the training of Meanformer with multi-scale features from an image-based backbone. Finally, a channel cross attention module is utilized to further improve the performance of DSTI by achieving cross-modal interaction. Experimental results show that the proposed method achieves advanced counting performance on six public datasets. Compared with the second-best method, the average reduction of MAE and RMSE reaches 5.4% and 5.5%, respectively.