Utilizing BERT for biomedical and clinical text mining

Biomedical and clinical text mining has always been an important but complicated task due to the complex nature of the domain corpora and the rapidly growing size of the documents. Recently, Bidirectional Encoder Representations from Transformers (BERT) has achieved great successes in a lot of natural language processing (NLP) tasks. Following these successes that have been made in these tasks, researchers in the medical research field started to apply BERT for improving the performance of the biomedical and clinical text mining models in the past year. Given that the fast changes and progresses have been made in this research field, in this chapter, we believe it is the right time to give a summary of the existing BERT models in the medical domain. Specifically, we classify these models into two groups, namely, pretrained BERT models and fine-tuned BERT models. We empirically compare the major contributions, architectures, datasets applied, and experiments conducted on these models; discuss the strengths and limitations of these models; and present the possible directions to deepen the biomedical and clinical text mining research with BERT in future.