Evaluation of Treatment Benefit in Journal of Clinical Oncology

Journal of Clinical Oncology is well-recognized for publishing manuscripts that describe treatment benefits or toxicities that have the potential to influence patient care. As such, it is critical to the editors and readers of our journal that such manuscripts present a high level of evidence that is not subject to bias or overstatement. In that regard, we have observed an increase in manuscripts that use observational study designs to assess benefit, partly due to the burgeoning interest in comparative effectiveness research (CER) coupled with easier access to large clinical and administrative databases that contain patient, treatment, and outcome data. One common example involves manuscripts describing the use of registry (eg, Surveillance, Epidemiology and End Results) and other administrative or clinical databases to analyze clinical outcomes of patients receiving different treatments. Although these studies use real world data that are often more representative of the variety of patients seen in clinical practice than is the case in randomized clinical trials (RCTs), there is a higher potential for bias and confounding with these designs, in part because treatment allocation is not randomized. Additional observational designs include time-trend studies which describe how treatment and outcomes have changed over time and attribute improvements in outcomes to more recent treatment approaches. However, because presentation, staging, concomitant care and other factors may also change over time, it is difficult to attribute these improved outcomes to any single factor, including recent treatment approaches. Likewise, modeling studies that use decision analytic or other approaches to quantify treatment benefits and harms or identify optimal treatment strategies in specific scenarios can be difficult to interpret with confidence. Many of the estimates and assumptions used in these models may not be valid, and models covering all contingencies may not be considered. As a result, the conclusions may not be correct. Thus, because of the potential for bias and confounding in observational studies, and because of potential inaccuracies in the assumptions and design features incorporated into modeling studies (despite the use of sensitivity analyses to examine the impact of variability in these estimates), these study designs are often sub-optimal to definitively demonstrate treatment benefits and harms. Given the potential limitations of these designs, in this editorial we would like to explain how we prioritize manuscripts submitted to JCO that claim to show a treatment benefit. The terminology used to describe treatment benefit can be confusing, but it is useful to make a distinction between two metrics, namely efficacy and effectiveness. By efficacy we refer to the outcome of a given treatment when administered under ideal circumstances (eg, in a defined population, with full compliance, delivered by competent physicians in a controlled environment, in the absence of comorbidity); in other words, whether an intervention works (or not) in a controlled situation. By effectiveness we refer to the outcome of a given treatment when administered in a more pragmatic (or real world) fashion, recognizing that compliance may be less than optimal, treatment settings may be diverse, expertise of care givers may vary and comorbidity may impact treatment outcomes. Typically, both efficacy and effectiveness are initially established in RCTs, which remain our gold standard for assessing treatment benefit. Meta-analyses that combine results of multiple RCTs can, at times, be useful to identify small(er) treatment effects that were not significant in individual trials but are clinically important, to examine overall treatment benefits when results of individual RCTs are conflicting, to explore patterns of treatment effects (eg, over time, in patient subsets) and to quantify rare toxicities. At JCO, meta-analyses that combine data at a patient level are prioritized over those that combine data at a study level, as they facilitate investigation of (and/or adjustment for) individual patient factors, and allow harmonization of analytic approaches and outcomes across studies. CER deserves special mention. We view research that investigates efficacy, effectiveness, and comparative effectiveness as a continuum, providing different but complementary information about treatment benefits and harms. CER that uses a randomized design is typically considered a form of effectiveness research and is evaluated at JCO in the same way as other RCTs. However, many CER studies use observational designs; they can be valuable to investigate patterns of harms and benefits of treatments in a variety of real world clinical settings, but they are susceptible to bias and confounding and do not reach the level of rigor associated with RCTs. For observational CER, JCO adopts the working definition put forward by the Institute of Medicine Committee: “CER is the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.” Key elements JOURNAL OF CLINICAL ONCOLOGY E D I T O R I A L VOLUME 31 NUMBER 9 MARCH 2