Assessing the Impact of Large Language Models on Cybersecurity Education: A Study of ChatGPT's Influence on Student Performance

The popularity of chatbots to facilitate day-to-day business, including students and their study exercises, is on the rise. This paper investigates the extent and effects on the academic performance of students that leverage such tools. While many other approaches are hypothesized and discussed, we measure empirically. We recorded and compared the performance of cybersecurity students in weekly exercises and final exams over a period of three years. This allows us to have three groups with varying degrees of ChatGPT influence, namely no access, uncontrolled access, and controlled access. In an anonymous survey, we found that approximately 80% of our students utilize ChatGPT during the weekly assignments in 2023. However, none of them indicated this on their submission, despite it being a mandatory requirement. Through statistical analysis of achieved points in our sample groups, we identified that students perform similarly on the weekly assignments. However, their performance on the final examination deteriorates.