An Intelligent Shadow Play System With Multi-dimensional Interactive Perception

We developed an intelligent shadow play system (ShadowTouch) based on multi-dimensional interactive perception technologies to let user interact with the shadow play figure naturally. ShadowTouch included a shadow play hardware subsystem and a multi-dimensional interactive software subsystem. In shadow play hardware subsystem, each joint of the shadow play figure was actuated by the connecting rod which was rotated through one steering gear. In multi-dimensional interactive software subsystem, a somatosensory interaction module synchronized motions between shadow play figure and human body based on skeleton point detection, and an automatic choreography module generated the motions automatically for the shadow play figure based on musical emotion recognition. The proposed system improved the immersive interaction experience and let users feel involved during the performance of shadow play, and the motions of shadow play figure (steering gear motion angles) could be saved digitally to further be applied for the reactivation and inheritance of traditional culture.