Summary: Research Diseases Need Holistic Care

Perhaps it is because for every action, there is an equal and opposite reaction. Perhaps it is human nature. For whatever reason, some situations solicit from us reactions that oscillate between extremes. If some vitamins are found to be counterfeit containing no nutritional value, many people react by swearing off the vitamin aisles of the local pharmacy. If a scene of electroconvulsive therapy in a popular movie is particularly disturbing (e.g., One Flew Over the Cuckoo’s Nest [1975]), many individuals will refuse this therapeutic option despite the evidence of its effectiveness and benefits.1 All too often, people react by overreacting. The same is true of our reactions to problems in research. In a recent publication, Antonakis2 argued that high-quality research in many domains is plagued by five diseases. Antonakis posits that these diseases threaten scholars’ ability to engage in high-quality research. They include: Significosis: a fixation on statistically significant results, Neophilia: an obsession with novel work, Theorrhea: a fetish for new theory, Arigorium: a lack of rigor in developing theory and undertaking empirical work, and Disjunctivitis: a penchant to accumulate large amounts of disparate factoids, salami sliced output, and trivial works, with little attempt at theoretical integration. On the whole, medical education researchers need to be mindful of these diseases because they can wreak havoc with the body of research in our domain. For instance, when applied to medical education: Significosis: The strong desire to publish statistically significant results can bias the true distribution of effect sizes, affecting how medical educators use results to inform practice and policy. Medical education researchers need to be mindful of the important distinction between statistical significance and educationally meaningful results. Neophilia: Obsessions with novel research and innovations can obscure the value of time-tested educational theories and processes. This can result in a blind dedication to novelty for novelty’s sake. Theorrhea: Reverence for continually developing or importing new theories into our field inhibits us from testing, refining, and confirming the theories we already have and use. Arigorium: Medical education researchers cannot develop findings and powerful insights without sufficient rigor in (1) the development of a theory (i.e., explicitly defining the limits of the theory, the assumptions and constructs it rests on, etc.) and (2) the conduct of empirical research (be that research quantitative—i.e., using a broad range of statistical methods appropriately, etc.—or qualitative—i.e., using methodologically aligned markers of rigor). Disjunctivitis: Disseminating multiple manuscripts, written by a crowd of coauthors, that offer small, perhaps even negligible contributions, does not advance collective knowledge in medical education. If anything, the sheer quantity of such publications makes it harder for scholars to wade through their depths to find articles that offer deep insights. Clearly these are diseases that should not be ignored lest they fester into truly toxic blights that pollute medical education’s research corpus. The Cures Can Be Worse Than the Diseases: The Need for Holistic Care In addressing these research diseases, we must be mindful not to impose cures that do more harm than good. By labeling these concerns as “diseases,” Antonakis implies that these concerns need to be cured. But completely eradicating these diseases (i.e., taking an extreme all-or-nothing stance) without critical reflection on the effect of those remedies poses its own problems. Take theorrhea as an example. Medical education’s journals often include descriptions of theories and/or theoretical constructs newly imported from other domains to be applied in empirical work. If we decide to stop importing, developing, and using new theories (i.e., to eradicate theorrhea through abstinence), then we risk losing significant and generative insights. For instance, cultural historical activity theory (CHAT) has supported the development of many important insights and practical applications for medical education—ranging from interprofessional collaboration3 to student learning goal development.4 If we decide to refrain from importing and developing new theories, these insights would be lost. However, if researchers only ever look to new theories, then medical education loses the ability to refine existing theories, or to reject long-standing theories through rigorous empirical testing. Therefore, we argue that the best cure for theorrhea is to encourage the inclusion of new theories in our scholarly efforts and to support the testing, application, and vetting of existing theories. This approach would support inclusivity. It would require medical education researchers to take a broad look at the field to see if a theory already reported in medical education can inform their research, or if the importing or developing of a new theory should shape their inquiry. The value of a balanced, holistic approach to addressing these research diseases is evident across the five diseases. We can even consider disjunctivitis—the disease perhaps most difficult to describe as a benefit and not just a blight. Antonakis2 notes that disjunctivitis is rooted in the academic incentivization of high numbers of peer-reviewed publications (i.e., that quantity of publication is more institutionally valued in things like promotion criteria than quality of publication). Antonakis states that disjunctivitis has contributed to the rise of research collaborations which he describes as “one failsafe way to boost [publication] output.”2 While Antonakis is careful to also note that coauthored papers are not inherently problematic, he also argues that research teams and journals have to be wary of free-rider problems and honorific authors.2 That notwithstanding, there is ample evidence and commentary from medical educators and researchers to indicate that collaboration between clinicians and PhD-trained scientists is a key element to the success of research efforts in our field.5 These collaborations are a strength of our domain and so should not be abandoned for fear of contracting disjunctivitis. While we must be certain that all authors on a manuscript meet authorship inclusion criteria, we must simultaneously respect the fact that our field has been built on collaboration that results in coauthorship. We need to preserve and protect that spirit of multidisciplinary collaboration, something that would be threatened by an overreaction to disjunctivitis. We contend that the best remedy for Antonakis’s five academic diseases is to engage in the full breadth of scholarly approaches. Statistical significance is important in identifying the precision of results, but it should not be the sole driving force in our studies. We need research that offers truly novel findings and that replicates findings and that reports on failed research. We need to lean on theories that have been well vetted and proven to be useful in medical education research and theories newly developed or newly imported from other domains to increase the scope of our thinking. We need new methods (or at least those not commonly used in medical education) to explore concepts and innovations as well as tried and well-established methods that continue to push our collective thinking and understanding. We need major leaps in thinking, as well as incremental adjustments in developing understanding. In short, we need a balanced, holistic reaction to address Antonakis’s research diseases. The 2017 RIME Supplement: An Illustration of the Holistic Approach This edition of Academic Medicine, the 2017 RIME supplement, includes manuscripts that reflect this balanced, holistic approach to research. Some articles illustrate how to avoid Antonakis’s research diseases, while others demonstrate why absolute avoidance of the diseases is not a panacea leading to good research. In their manuscript about emancipatory medical education, Zaidi et al6 use the theory of emancipatory pedagogy (a relatively new theory to enter into medical education’s discourse) and critical discourse analysis to investigate educators’ experiences of facilitating cultural discussions in global health professions education programs, and the lessons those educators learned about critical consciousness. While this paper could be diagnosed with theorrhea, it is the theory used in the paper that enables the collaborative team to construct important insights into educators’ experiences of working with students who survived traumatic experiences of racism and power differentials. We would suggest that part of the contribution this manuscript makes to our field is the use of the theory of emancipatory pedagogy to support the critical analysis of medical education in creating a more just society by having educators and learners work as transformative intellectuals. Bowen and colleagues’7 study of physicians’ motivations for following up on patients after transitioning responsibility for the patient to another physician uses self-determination theory (SDT) to inform the discussion of their results. In contrast to Zaidi and colleagues’6 use of a “new” theory, SDT is a theory that has been used with some regularity in medical education research. In this study, the authors use SDT to make sense of participants’ personal attachment to patients as a motive for following up after transition. They note that electronic health records, a technology that was not available when SDT was developed, can enable physicians to follow up on patients with whom they have connected. This research further develops SDT by suggesting a “technology-mediated cultural evolution” to the meaning of SDT’s concept of “relatedness.” Kindler and colleagues8 describe how, in a problem-based learning environment, preclinical medical students set learning goals and assess their achievement of those goals. Instead of introducing a theory that is unfamiliar to the medical education community, the authors avoid theorrhea by using an existing theory of self-regulated learning as a background for their examination of students’ beliefs about setting and achieving their learning goals. Their results demonstrate a tension between existing curricula, assessments, and what students perceive to be effective assessments of their learning. While students varied considerably in their methods for setting goals, a common theme they described was clinical application and preparedness as a measure of achievement. In fact, the authors describe how a lack of clinical exposure actually served as a barrier to students for effective prioritization and goal setting. This work raises questions about the timing and quality of clinical experiences and its relationship to students’ goal setting and monitoring, but also helps to further the discussion about self-directed learning—a theory that is regularly used in medical education research. In their investigation of the conflicts experienced by interprofessional health care teams working in clinical contexts, Bochatay et al9 engage in a foundational exploratory study, through which they develop a framework for understanding workplace conflicts. The model addresses the complexity of conflict, across individual, interactional, and organizational levels, and suggests how consequences of and responses to conflict create cycles of continued tension and conflict. This new model, which builds on and extends previous frameworks, suggests causal relationships that can be tested in future research. This manuscript illustrates how new theoretical constructs, that build on existing research, can support the development of new foundational knowledge and inform practical educational interventions. We’d lose these gains if we refused to import or create new theories—that is, if we cured theorrhea by abstinence. The study by Plant et al10 examines how faculty and residents use reflection to continuously improve patient care in authentic clinical settings. The authors argue that there is no clear understanding of reflection, despite prevalent use of the concept. The study contributes to the literature by providing a clearer operationalization of critical reflection and how it promotes transformative learning. The authors defy theorrhea by advancing our understanding of critical reflection. Data were gathered from three pediatric residency programs through six focus groups. Through careful qualitative analysis, the authors found that reflection occurred more commonly on action (retrospectively), and less commonly in action (concurrently in action). They also noted that when reflection occurred, it led to critical reflection, if goal and action were directed. The authors propose a model for explaining how critical reflection can occur, but do this by carefully linking their findings to existing literature and aligning their implications with theory. Burk-Rafel and colleagues11 investigate factors that contribute to Step 1 performance. The authors argue that factors are “modifiable” (such as study strategies and test-preparation resources), and so this study seeks to advance theory and empirical evidence in learners’ self-directed parallel curriculum. Frameworks from practice and spaced repetition may help explain the impact of some modifiable study strategies, yet a deeper understanding of learners’ test preparation behavior has not been previously reported. The authors avoid arigorium in pursuing this work by presenting a rigorously designed statistical analysis of medical student survey data collected across two years with Step 1 performance. They report on study hours, study resources, and self-reported study goals. These are modifiable behaviors which can help educators and learners maximize their test preparation and performance. The authors should also be commended for their attention to detail in describing their statistical methods, summary of data, and presentation of results. Such clarity is a strong defense against arigorium. The study by Abbott et al12 examines the impact of simulation and feedback, based on either expert task demonstration or personalized video feedback. The authors used general surgery interns and research fellows to measure their performance and perceived workload. The controlled trials implemented by the authors demonstrate methodological rigor in their experimental design and analysis that used historical controls—a design that is clearly not a case of arigorium. The authors found that personalized video feedback improved performance and reduced workload, adding to empirical evidence on our understanding of skill acquisition and retention. As a potential solution to a particular case of arigorium, Taylor and colleagues13 have introduced a novel rubric to assess the increasing number of entrustable professional activities (EPAs) used widely for UME and GME trainees. While critical for competency-based medical education, some EPAs have been introduced which do not conform to literature-described standards. The EQual rubric was developed by the investigators in a rigorous process that involved review of the literature, development of an instrument with criterion-based descriptive anchors, expert consensus and review, and, finally, training and instrument testing. Applying their rubric to existing EPAs across specialties, the EQual tool provided reliable measurement of the EPAs relative to standards, across both program directors and research assistants at multiple sites. Not only does this tool help to evaluate the quality of the current EPAs in use, but it will also help in evaluation of those which are questionable and in development of new EPAs for assessment. Asgarova et al14 explore how medical students experience patient continuity during longitudinal clinical placements. The methodology and methods used in this paper are common qualitative approaches. The authors are transparent about their methods, making their processes easily evaluated by the reader. Indeed, this manuscript is a nice example of a small-scope study, conducted using a clear qualitative research design, and informed by a theoretical framework that delivers valuable insights for the community. In defiance of arigorium, it illustrates the contributions that can be generated with a clear research design that relies on tried-and-tested methods. In their work, Cendán et al15 describe the feasibility and impact of a novel mobile application for the real-time assessment of professionalism-related behaviors in the clinical environment. In their mixed-methods approach, survey-based data were analyzed to assess the technology’s functionality and qualitative methods were used to explore the user experience with the application. Their application, PROMOBES (Professional Mobile Monitoring of Behaviors), was used by faculty to assess trainees at two sites around six domains of professionalism. This research is rooted in a deep exploration and development of the professionalism framework used; and so, while it reports on a new innovation, it is grounded in a preexisting framework and so avoids neophilia. Stojan et al16 examine the impact of diagnostic uncertainty on medical student decision making in the standardized patient environment during a summative OSCE. Fourth-year medical students were provided with a case scenario and various levels of pretest probability for a comorbid condition and were asked to decide about treatment, no treatment, or further testing to confirm the diagnosis. Regardless of the pretest probabilities provided, most students chose to pursue additional, and perhaps unnecessary, diagnostic testing. The authors’ methodology is well rooted in accepted theoretical models of decision-making thresholds and examines the concepts of high versus low testing thresholds, and their implications for cost-conscious and safe patient care. The manuscript provides an example of using a time-tested methodology (OSCE-based assessment) to test a novel theory (decision-making confidence) with widespread application for trainees across the learning continuum. In this blending, the authors defy neophilia while still engaging in novel work. The study by Peterson et al17 demonstrates how to avoid neophilia and disjunctivitis. The authors build on previous results to replicate and generalize their findings to multiple institutions. The authors study two surveys—Readiness for Clerkship (RfC) and Readiness for Residency (RfR)—administered across four medical schools and two academic years to measure the readiness of learners halfway through medical school and at the time of graduation. The surveys were developed using the CanMEDs framework and previously examined at a single institution. In this study, the authors use multisite data to generalize their results on the precision of survey results and evaluate whether the strengths and weaknesses of learners are unique to specific schools, providing meaningful comparisons. The authors confirm previously reported findings on the reproducibility of the surveys as well as identify differences in school curricula that result in learner preparedness; for example, a school emphasized clinical skills development, while another used learning plans to help clerkship students. Faculty members are concerned with at-risk students and aim to identify early signs of difficulty so that learners can be remediated. The approach developed by Brenner and colleagues18 presents a methodologically rigorous and innovative model for predicting at-risk students. The authors use learner performance from NBME custom questions to predict their performance on Step 1, while controlling for other confounding factors. Data were taken from three classes of students to develop a regression model. To provide evidence on the predictive accuracy of the statistical model, the authors conduct cross-validation studies. This demonstrates evidence of predictive power, which is often lacking in studies that are focused on significosis. For example, the authors show the false-positive and false-negative rates of their model. The authors also provide meticulous detail to their methods and results, which can easily be replicated at other institutions. The manuscript by Park et al19 avoids arigorium by conducting a multisite collaborative study to establish validity and reliability of scoring guidelines used for clinical graduation competency examinations, assessing both clinical skills and the written progress note. Standardized cases, scoring rubrics, and rater training were shared and performed at seven medical school sites. The authors found that assessment of student performance varied by the task performed and the clinical scenario. School effect was found to account for only 1% of the variance between schools. However, more variability was noted in the progress note rating among schools. These findings certainly highlight the need for more collaborative efforts and assessment among schools and also highlight potential differences in approaches to teaching clinical reasoning. Conclusion Research in medical education can fall prey to the diseases that plague other areas of inquiry. We must avoid extreme reactions that can make the cure worse than the afflicting diseases. Medical education is a field of scholarly inquiry where theories from multiple disciplines inform and are informed by practice. Foundational research in medical education often involves deeply exploring the breadth of topics related to the education of future health care professionals: for instance, professionalism, assessment of competence (be it individual or collective focused), and team collaboration. Scholars then take these foundational findings and translate them into practice contexts: for instance, creating mobile technologies for assessing professionalism in situ,15 creating rubrics for the evaluation of EPAs,13 and exploring the conflicts experienced by interprofessional teams so that we can generate effective conflict management training programs.9 In so doing, medical education scholars use the knowledge, methods, and theories developed in other fields to inform our practice, and we contribute our own insights back to these fields by refining ideas, revising theories, or developing new frameworks. We also learn from our own experiences and scholarly efforts, and so build theories, knowledge, and best practice guidelines for our community—forever constructing better understandings, uncovering hidden assumptions, and testing new ideas. This multidisciplinary work is truly a collective effort. Working across disciplinary boundaries—when PhDs from many different disciplines work with MDs from a wide range of specialties—we strengths and This helps to make medical education a powerful field of and illustrates what can when theory practice, and when disciplinary are a and holistic approach, we can the power of our the where theory practice, and when disciplinary are avoid extreme this approach, the cure to research diseases the potential and power of collaborative and rigorous research to advance medical education.