Action recognition based on RGB and skeleton data sets: A survey

Action recognition is a major branch of computer vision research. As a widely used technology, action recognition has been applied to human–computer interaction, intelligent pension, and intelligent transportation system. Because of the explosive growth of action recognition related methods, the performance of action recognition on many difficult data sets has improved significantly. In terms of the different data sets used for action recognition, action recognition can mainly be divided into RGB-based action recognition method and skeleton-based action recognition method. The former method can take advantage of the prior knowledge of image recognition. However, it has high requirements for computing power and storage ability, and it is difficult to avoid the influence of irrelevant background and illumination. In contrast, the latter method’s calculation amount and required storage space are reduced significantly. However, it lacks context information that is useful for action recognition. This review provides a comprehensive description of these two methods, covering the milestone algorithms, the state-of-the-art algorithms, the commonly used data sets, evaluation metrics, challenges, and promising future directions. So far as we know, this work is the first survey covering traditional methods of action recognition, RGB-based end-to-end action recognition method, pose estimation, and skeleton-based action recognition in one review. This survey aims to help scholars who study action recognition technology to systematically learn action recognition technology, select data sets, understand current challenges, and choose promising future research directions.