A Novel Approach for Text Classification by Combining Pre-trained BERT Model with CNN Classifier

Language model pre-training has emerged as a highly effective approach for acquiring universal language representations. Among the state-of-the-art models in this field, BERT (Bidirectional Encoder Representations from Transformers) has garnered significant attention due to its remarkable performance across various language understanding tasks. With its advanced architecture and innovative techniques, BERT has consistently delivered impressive results, solidifying its position as a leading language model pre-training model. CNN (Convolutional Neural Network) is also widely used in language model training.This study presents a novel approach for text classification by combining a pre-trained BERT model with a CNN classifier. The model under consideration attains the highest level of performance currently available on the AG News and Amazon product reviews datasets. The BERT model is applied to produce contextualized embeddings for the input text data, while the CNN classifier executes the classification process. By utilizing the contextualized embeddings of BERT and the n-gram feature capturing of CNN, our model exhibits superior performance in both accuracy and F1 score when compared to the baseline models. The findings indicate the capability of our methodology in analyzing social media, determining sentiment, and classifying documents.