Assessing the Statistical Abilities of GPT-4 as a Zero-shot Reasoner: Strengths and Weaknesses

Abstract Background The field of artificial intelligence (AI) has witnessed remarkable advancements in natural language processing and understanding (NLP and NLU), particularly through the development of large language models like GPT-4. These models have demonstrated promise across various domains, including medicine. However, the extent to which they can reason statistically as zero-shot reasoners remains largely unexplored. Objective This study aims to evaluate GPT-4's statistical abilities as a zero-shot reasoner and identify its strengths and weaknesses in addressing statistical inquiries. Methods A diverse set of statistical questions was extracted from published articles and inputted into the GPT-4 chatbot. The responses generated by GPT-4 were assessed for accuracy and compared against the correct answers. Logistic regression analyses were performed to determine the impact of statistical topics and AI tasks on the model's performance. Results GPT-4 achieved an overall accuracy of 74% in responding to the statistical questions. The analysis revealed that the model excelled at comprehending well-structured and unambiguous questions, showcasing proficiency in entity recognition and information integration. However, it encountered difficulties with complex statistical concepts, data interpretation, and questions involving ambiguity or convoluted structures. Conclusion GPT-4 demonstrated strengths in understanding statistical concepts when presented clearly. However, it faced challenges with more intricate tasks and synthesizing information from multiple sources. Suggestions for improvement include fine-tuning the model using advanced statistical datasets, incorporating external knowledge sources, optimizing prompt engineering techniques, and enabling visual information processing. Implications: This study offers insights into the strengths and weaknesses of GPT-4 as a zero-shot reasoner in statistical tasks. Although the model exhibits potential in basic statistical reasoning, caution should be exercised in relying solely on its responses without human supervision for comprehensive statistical analysis and interpretation. Additionally, given that most researchers in the medical field may lack statistical expertise, leveraging language models can be valuable for addressing their statistical inquiries.