A Nanoparticle-Based Artificial Ear for Personalized Classification of Emotions in the Human Voice Using Deep Learning

Artificial intelligence and human-computer interaction advances demand bioinspired sensing modalities capable of comprehending human affective states and speech. However, endowing skin-like interfaces with such intricate perception abilities remains challenging. Here, we have developed a flexible piezoresistive artificial ear (AE) sensor based on gold nanoparticles, which can convert sound signals into electrical signals through changes in resistance. By testing the sensor's performance at both frequency and sound pressure level (SPL), the AE has a frequency response range of 20 Hz to 12 kHz and can sense sound signals from up to 5 m away at a frequency of 1 kHz and an SPL of 126 dB. Furthermore, through deep learning, the device achieves up to 96.9% and 95.0% accuracy in classification and recognition applications for seven emotional and eight urban environmental noises, respectively. Hence, on one hand, our device can monitor the patient's emotional state by their speech, such as sudden yelling and screaming, which can help healthcare workers understand patients' condition in time. On the other hand, the device could also be used for real-time monitoring of noise levels in aircraft, ships, factories, and other high-decibel equipment and environments.