ChatGPT in head and neck scientific writing: A precautionary anecdote

To evaluate the accuracy of ChatGPT references in scientific writing relevant to head and neck surgery. Five commonly researched keywords relevant to head and neck surgery were selected (osteoradionecrosis of the jaws, oral cancer, adjuvant therapy for oral cancer, TORS, and free flap reconstruction in oral cancer). The AI chatbot was then asked to provide ten complete citations for each of the keywords. Two independent authors reviewed the results for accuracy and assigned each article a numerical score based on pre-selected criteria. Among 50 total references provided by ChatGPT, only five (10 %) were found to have the correct title, journal, authors, year of publication, and DOI. Merely 14 % of the presented references had correct DOI. References regarding free flap reconstruction for oral cancer were the least accurate from all the five categories, with no correct DOI. Complete inter-rater agreement was noted while evaluating the citations. Only 10 % of the articles provided by ChatGPT, relevant to head and neck surgery, were correct. A high degree of academic hallucination was noted.