PianoWatch: An Intelligent Piano Understanding and Evaluation System Using Smartwatch

Existing intelligent piano learning systems mainly assist the player by camera, which generally only consider the fingering that can only reflect the performance problem from a limited perspective. Thus, we propose PianoWatch, a multi-dimensional assistance system based on wrist wearable devices. PianoWatch extracts more accurate patterns by analyzing data from microphone, camera, accelerometers and gyroscopes. Then it gives corresponding playing advices, in addition to pitch, including fingering, depth and even mood, through an evaluation model. We implement the prototype of PianoWatch and evaluate it by 20 volunteers. Extensive experiments show it can achieve 93% F1-Score on the playing pattern extraction, and more than 95% of users would like to try it to assist their piano learning.