Promptcblue: A Chinese Prompt Tuning Benchmark for the Medical Domain

Biomedical language understanding benchmarks are the driving forces for artificial intelligence applications with large language model (LLM) back-ends. However, most current benchmarks (a) are limited to English, which makes it challenging to replicate many of the successes in English for other languages, or (b) focus on knowledge probing of LLMs and neglect to evaluate how LLMs apply these knowledge to perform on a wide range of bio-medical tasks, or (c) have become a publicly available corpus and are leaked to LLMs during pre-training. To facilitate the research in Chinese medical LLMs, we rebuilt the Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark into a large-scale prompt-tuning benchmark, PromptCBLUE. Our benchmark is a suitable test-bed and an online platform for evaluating Chinese LLMs' multi-task capabilities on various bio-medical tasks, including medical entity recognition, medical text classification, medical natural language inference, medical dialogue understanding, and medical content/dialogue generation. To establish evaluation on these tasks, we have experimented and reported the results with the current 9 Chinese LLMs fine-tuned with different fine-tuning techniques.6 Our benchmark is released at https://tianchi.aliyun.com/dataset/165438.