Music to Dance: Motion Generation Based on Multi-Feature Fusion Strategy

Studies on generating dance sequences from music can greatly promote dance teaching and animation production. However, the current results of such tasks are not very satisfactory. Due to the lack of consideration of human body structure, unreal phenomena such as slippery feet and twisted joints will appear in the resulting movements. Since only a single correlation feature between music and dance is learned, there is a problem of poor matching between generated actions and music. To solve these problems, we propose a learning framework based on GAN and a multi-feature fusion strategy to realize the mapping from music to dance. We use two discriminators to constrain style and authenticity respectively to make our actions generated by the network more coherent and natural, which is essential in producing authentic dance sequences. Moreover, we extracted the three characteristics of music style, beat, and structure. Then we used the feature fusion method to obtain a comprehensive feature representation, which guarantees the consistency of the generated dance sequence with the input music to the greatest extent. The experimental quantitative and qualitative results show that our method can generate accurate, consistent, beat-matching dance movements from music. In addition, we also collected and produced a data set containing music features and corresponding pose sequences, which is convenient for pose generation based on music features.