Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis

BACKGROUND: Expert feedback lays the foundation of rigorous research. However, the rapid growth of scholarly production challenges the conventional scientific feedback mechanisms. High-quality peer reviews are increasingly difficult to obtain. METHODS: . Additionally, we conducted a prospective user study with 308 researchers from 110 institutions in the fields of artificial intelligence and computational biology to understand how researchers perceive feedback generated by our system on their own papers. RESULTS: portfolio show similar patterns. In our prospective user study, more than half (57.4%) of the users found GPT-4-generated feedback helpful/very helpful, and 82.4% found it more beneficial than feedback from at least some human reviewers. We also identify several limitations of large language model (LLM)-generated feedback. CONCLUSIONS: Through both retrospective and prospective evaluation, we find substantial overlap between LLM and human feedback as well as positive user perceptions regarding the usefulness of LLM feedback. Although human expert review should continue to be the foundation of the scientific process, LLM feedback could benefit researchers, especially when timely expert feedback is not available and in earlier stages of manuscript preparation. (Funded by the Chan-Zuckerberg Initiative and the Stanford Interdisciplinary Graduate Fellowship.).