Foreword: Sticks and Stones: Moving Toward a More Productive Peer-Review Process for Authors and Reviewers

On behalf of the Research in Medical Education (RIME) Program Planning Committee (PPC), welcome to the 56th annual Association of American Medical Colleges (AAMC) RIME program. We believe you will appreciate the innovation and relevance of the 14 research papers as well as the Commentary and Summary pieces included in this Academic Medicine supplement and the RIME-sponsored sessions during Learn Serve Lead 2017. We thank all the researchers who submitted manuscripts for consideration—it was a privilege and honor to review your work. We also thank the 58 reviewers who participated in the peer-review process—without you, there would be no RIME program. As a committee, we encourage you to consider the studies or reviews you now have under way in terms of their potential as submissions to future RIME programs. As investigators, we often begin research by envisioning the outcome and where we might submit our findings. Envisioning your work as a RIME presentation and article may help provide context and a timeline. In regard to context, we are interested in accepting a broad range of manuscripts that report innovations in teaching, learning, and assessment that are generalizable and advance the field of medical education. We are particularly interested in manuscripts that report solutions to the challenge of attracting and supporting learners and faculty who are underrepresented in medicine as there is still far too little research in that area.1 With regard to timeline, we are delighted that the deadline for future RIME submissions will be mid-December for the foreseeable future. We believe having a consistent deadline going forward will help with planning. So we ask you—what problems and innovations in medical education are you studying? Even as early as the design phase, please consider whether your work might make a great RIME submission. The 2016 RIME Foreword outlined the elements of a successful RIME submission.2 This year’s RIME Foreword is intended as a companion piece that considers the process of peer review for medical education research and the ways in which we, as investigators, respond to it. While our observations are not focused solely on the RIME process, RIME is a rich microcosm from which we can all learn. The Multiple Roles of the RIME Reviewer The RIME review process hinges on the work of a select group of peer reviewers that take the first pass at assessing the quality of submitted manuscripts. It is important that reviewers understand that when reviewing manuscripts, their work has two distinct purposes: to advise the RIME PPC on whether a manuscript should be accepted to RIME and to provide constructive feedback to all authors regardless of the final decision on their submission. To effectively address both of these purposes, we have found it useful to keep in mind that the manuscript is not the study, but rather it is a description of the study. This distinction is important because it serves as a useful framing perspective for providing comments. Suppose a reviewer notices a gap in the description of methodology (a common error because we sometimes get so close to our own work that we overlook the obvious), or questions the choice of a particular method or analytic process. Understanding that the manuscript is not the study allows the reviewer to provide constructive observations about what might be missing or needs greater clarification without assuming an error in the study itself. For example, the comment that “the authors need to better justify the atypical use of a G-study” is a focused observation that provides actionable feedback, while the comment that “I believe the G-study analysis was misapplied” is judgmental and makes assumptions about the study itself. Why assume an error in methodology when the real problem may be a gap in communication? We on the RIME PPC have tried to push ourselves to be open to studies that use a variety of valid analytic methods (e.g., quantitative, qualitative, or mixed methods), as long as they are selected to achieve the goal of answering the study question. For some research questions, this may require adopting research methods more often found in social sciences than in medical education and the citation of literature outside of our own discipline. Just because a technique is new to us does not mean that it is wrong, and we believe this philosophy will serve us well as medical education research reaches out into new and perhaps more sensitive and complex topics. What the Literature Tells Us About Being a Reviewer Looking beyond RIME’s specific process, the literature on being an effective peer reviewer falls into two general groups: articles calling for civility in reviews, and articles on the need for reviewers to have content-specific knowledge and training in the process of review. On the need for civility, both MacInnis3 and Schneiderhan4 take a lighthearted approach to delivering a serious message. MacInnis takes us back to kindergarten by suggesting that reviewers need to play well with others, and Schneiderhan quotes Taylor Swift: “Why you gotta be so mean?” It accomplishes nothing to antagonize an author, and it makes an editor’s job more complicated when they must tidy up snarky comments before providing useful feedback to the author. In an effort to identify what makes a good peer reviewer, Callaham and Tercier5 explored career variables such as length of experience and serving on an IRB to find possible correlations to the quality of reviews produced. Interestingly, they found “no easily identifiable types of formal training or experience that predict reviewer performance.” Despite these findings, we speculate that there may be subtle intrinsic variables in play that make reviewers successful. For example, in being an effective teacher, it has been shown that being invested in learner success supersedes knowledge of theory.6,7 In a similar way, perhaps being an effective reviewer requires one to be invested in the author’s ultimate success by providing specific, actionable feedback that leads to eventual publication, even if not in that specific journal. The work of other investigators suggests that being invested in the author’s success, by itself, is not enough. For example, Sullivan et al8 took a training approach to the need for well-prepared reviewers and created an overview of a curriculum to build a specific set of reviewer skills. To do this, they identify the five key duties of a reviewer as ensuring that (1) the paper adds value to the field, (2) the journal’s mission is upheld, (3) the principles of scientific process and ethical standards are met, (4) the methodology is sound, and (5) feedback given to authors is constructive. Spigt and Arts9 took a different approach by looking at reviewing in a holistic way and performing a task analysis of the review process (see their Table 1). They encourage reviewers to carefully consider whether they have the time and expertise to do the job before accepting the review, and they provide well-written examples of ways to comment or make inquires to help authors clarify their work. In summary, the reviewer’s role is complex and serves several purposes. Reviewers assist editors (or selection committees) by identifying articles that are appropriate for a journal or event, providing an independent assessment of the merit of the manuscript and a sense of how much additional work would be needed to get the manuscript ready for press, and identifying work that might not be suitable for publication in a specific journal. Reviewers also assist authors by providing specific, actionable, constructive feedback to help authors refine the work into publishable form. What the Literature Tells Us About Responding to Review Even those who are a bit weary of teaching with acronyms will appreciate Agarwal et al10 as they discuss how to REAP REWARDS from reviewers’ comments. This article takes an engaging, conversational tone in working through the steps an author can take when a review is received. Whether the decision is the dreaded rejection or the more hopeful revise and resubmit, the REAP REWARDS approach takes the sting out of reviews by providing authors a plan of action to improve the work. The process is: Read the reviews, Emote, Arrange reviewers’ comments, and Parse responsibility. Then: Revisit the manuscript, Evaluate each comment, Write responses, Argue amongst yourselves, Rewrite the manuscript, Direct reviewer attention to responses, and Submit the revised manuscript. This article may be especially beneficial to early-career researchers and authors who struggle to put review comments into context. Specific to medical writing, both Cummings and Rivara11 and Williams12 provide suggestions for how to respond to reviewers’ comments. The central idea is that the author should respond to the editor rather than the reviewer who made the comments, so the reviewer becomes the third party. The author’s job is to respond respectfully and thoughtfully to each comment, to inform the editor what action was taken, and finally to make the change/correction in the revised manuscript (see Figure 1). Building on this concept, Kotsis and Chung13 provided a decision chart (see their Figure 1) to help authors make logical decisions about the next steps in responding to a review and how to consider the probability of success. Breaking the task down into steps not only makes it feel less onerous but also serves as a good distraction until our hurt feelings are soothed.Figure 1: Review and revise feedback system.While some journals provide a template for responding to reviewers, putting comments and responses into a two-column table and numbering each is a simple system to organize the process. When reviewers do not agree, a reasonable strategy is to explain to the editor which of the specific comments best aligns with your own strategy and take that corrective action. Of course, there will be times, perhaps many times, when authors disagree with comments or suggestions from a reviewer. In general, arguing with reviewers’ comments when responding to editors is not constructive. When we have been in that situation, we have found that the most constructive approach is to try to accept the intent of the comments as constructive feedback. In writing research manuscripts, as in life, resisting informed feedback usually ends up being counterproductive. After all, if a reviewer misunderstands what we have written, the chances are that at least some readers will also misunderstand. Even if we think a reviewer’s comment is just plain wrong, the real message is that our writing likely needs refinement. Conclusions As medical education research expands to embrace new and more complex questions about learners, teachers, the learning environment, and patient care, we must be ready with effective strategies to review and revise this work for dissemination. When we delve into content areas that are challenging, it becomes even more important for researchers to be rigorous, and reviewers to be diligent in ensuring the writing reflects a complete, unbiased, and valid process. The reviewers’ job is usually anonymous to all but the editor, but the reviewer is entitled to a sense of pride when they read an article that they helped move toward publication. They call it “peer review” for a reason; we are an academic community with similar interests and goals. So mutual support in the process of medical education research benefits us all in the short term and, we believe, benefits patients in the long term. Gratitude We thank several individuals and groups who helped make this year’s RIME program a reality. First, we thank the dedicated AAMC staff—Nesha Brown, Steve McKenzie, and Kate McOwen—who worked tirelessly to make the RIME program and supplement happen. We also thank the incredible scholars who join us on the RIME PPC: Bonnie Miller, Lara Varpio, Tanya Horsley, Bridget O’Brien, Win May, Jeanne Farman, and Yoon Soo Park. It is an honor and privilege to work with such a talented group of individuals who are so devoted to promoting the art and science of medical education. Finally, we thank Dr. David Sklar and the editorial team of Academic Medicine. We sincerely appreciate their continued support of medical education research and the RIME program.