A Text Classification Model Based on BERT and Attention

Text classification, as an important task in the field of natural language processing, aims to classify text data into specified categories in order to mine valuable information from the huge amount of text data. To address the problems that the current deep neural network-based text classification methods have inadequate feature extraction, poor semantic representation, and sensitivity to sentence length, this paper proposes a text classification model based on BERT and attention mechanism. The model is based on BERT, which automatically learns the feature vector representation of text data, and then further fits the semantic features by a neural network with residual connections. Finally, the attention mechanism is used to learn the importance of different features adaptively and weight the representation to improve the performance of text classification. It is shown through experiments that the model can effectively mine the deep feature information of the text and can adapt well to the problem of sentence length differences and uneven sample distribution. The accuracy and F1-score s on the publicly available SST-2 dataset reach 92.66% and 92.92%, respectively, with excellent classification results.