Abstract A new emotion recognition system based on speech is constructed to improve the ability of recognizing negative emotions. Multi-dimensional acoustic characteristics were tested and among them, short-term energy and Mel-frequency cepstral coefficients (MFCC) were selected to be used as parameters for recognition. The system consists two modes: single recognition and group recognition. Single recognition adopts BP neural network model based on MFCC, while group recognition adds support vector machine model based on short-term energy on the basis of single recognition which the group recognition rate of 20 speech can reach 97%. With the increase of the number of speech in each group, the recognition accuracy of negative emotion tends to 100%.