What Does It Take to Transform Real‐World Data Into Real‐World Evidence?

The use of real-world data (RWD) to generate real-world evidence (RWE) to inform certain regulatory and healthcare-related decisions is not a new concept. However, in recent years, an increased potential to utilize RWD to generate evidence to inform healthcare and drug development decisions (Figure 1) has been identified for (i) developing decision-support tools and practice guidelines, (ii) supporting coverage decisions, (iii) monitoring postmarket safety and adverse events of approved therapeutic products as well as supporting efficacy of therapeutic products, and (iv) supporting clinical trial designs (e.g., pragmatic trials) to develop innovative treatments. In the United States, the uptick in RWD-and RWE-related conversations and collaborations can be attributed to a number of legislations that were signed into law.1-3 The 21st Century Cures Act (2016) required the US Food and Drug Administration (FDA) to establish a program to evaluate the potential use of RWE to support (i) approval of a new indication for a drug approved under section 505(c) and (ii) postapproval study requirements. Subsequently, the sixth Prescription Drug User Fee Act (PDUFA VI; 2017) also required the FDA to initiate appropriate activities (e.g., pilot studies or methodology development projects) to address key issues in using RWE to make regulatory decisions. In part, the Health Information Technology for Economic and Clinical Health (HITECH) Act (2009) that promoted increased adoption of electronic health records (EHRs) has also led to increased collection of RWD in an electronic format. RWD are data related to patient health status and/or the delivery of health care that are routinely collected from a variety of sources, such as EHRs, claims and billing data, medical product and disease registries, patient-generated data including in home-use settings, and data gathered from other sources that can inform on health status (e.g., mobile devices).4 RWE is the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD.4 "If you torture the data long enough, it will confess to anything" However, RWD can be "noisy" when compared with data collected from RCTs. For example, the rigor of RWD collection methods may not be comparable to that of clinical trial data collection methods. In RCTs, randomization is a critical tool to balance known or unknown variables that may bias clinical outcomes. However, RWD are collected in the real-world setting of heterogeneous patient populations and clinical practices. The noise associated with RWD can come from recorded or unrecorded confounders (e.g., patient features, changes in clinical practice by geography or over time, payers' preference, physician's decision or options to select a drug or dose, concomitant medications, medication history, and comorbidity) leading to certain biases in RWE generation. Additionally, RWD sources (e.g., EHRs, health claims, and safety reports) have their own unique features, designs, strengths, and limitations. As the original purpose of these RWD sources vary, they may not always capture all the data elements needed to answer a specific research or regulatory question. Furthermore, accessing and linking multiple data sources (e.g., medication history data from a payer and survival data from another source) to obtain comprehensive, longitudinal information on the same patient can be challenging. As a result, there are many logistical, technological, methodological, legal, and privacy challenges in realizing the full potential of RWD. The articles in this themed issue highlight some of the progress, challenges, and potential solutions for converting RWD into RWE that is fit for decision making. RWD has been used to generate RWE to monitor postmarket safety or to provide data to support efficacy during regulatory approval of therapeutic products. As Dal Pan5 describes, the FDA has used RWD primarily in the evaluation of safety via efforts such as Sentinel Initiative to make many regulatory decisions, including in some instances, eliminating the need for an industry-sponsored postmarketing study. Only in a few instances has RWD provided evidence to support efficacy of therapeutic products. This has primarily been in the setting of oncology and rare diseases (e.g., as historical controls), where there is a high unmet need (Table 1). Much of the FDA's experience with RWD and RWE comes from the regulation of devices. This is because the very nature of medical devices makes the implementation of RCTs challenging. Fleurence and Shuren6 describe the multistakeholder initiative National Evaluation System for health Technology (NEST), which aims to improve evidence generation for medical devices to inform decisions across the total product lifecycle, including marketing authorization, postmarket surveillance, payer coverage and reimbursement, clinical practice, and patient choice. Several test cases are being evaluated and are expected to provide important practical and scientific learnings that can inform future studies. It is reasonable to expect that RWE can support regulatory decision making for indication expansion and safety monitoring for vaccines. For example, the FDA's Center for Biologics Evaluation and Research (CBER) recently updated the label of zoster vaccine live (Zostavax; Merck, Whitehouse Station, NJ) based on data from the interim analysis of a prospective observational study to support longer-term effectiveness in individuals older than 50 years.7 To address a regulatory question, both data and analyses should be reliable, fit, and reproducible to ensure generation of regulatory-grade RWE.8 Advancing RWD collection to generate regulatory quality RWE is a key strategic priority for the FDA. As ElZarrad and Corrigan-Curay9 discuss, the recently published framework4 highlights the opportunities for using RWD to develop evidence that can support regulatory decision making. The framework includes considerations for (i) whether the RWD are fit for purpose in regulatory decision making, (ii) whether the methodologies used to generate RWE can provide adequate scientific evidence to address the regulatory questions presented, and (iii) whether the approach used in a particular case meets the FDA's regulatory requirements. Additional FDA guidances are expected to be published in the coming years. The intention of these guidances is to advance utility of RWD to generate evidence that can better inform regulatory decision making. Cave et al.10 highlight experiences from a European perspective. The European Medicines Agency (EMA) uses RWD for safety signal evaluation, risk management, and lifecycle benefit-risk evaluation.10 Like the FDA, the EMA has less experience in using RWD for supporting efficacy decisions. To support efficacy, the EMA has used RWE for the approval of gene therapy products as well as chimeric antigen receptor T-cell therapy products, somatic cell therapy products, and antisense oligonucleotides (Supplementary Table of ref. 10). Additionally, as data from patient registries evolve to generate high-quality data to support regulatory decision making, Olmo et al.11 describe some of the barriers and opportunities identified from the EMA's Patient Registry Initiative that may enable a more systematic approach for utilizing registry-based RWD to generate RWE. Although the EMA and the FDA share multiple technical (e.g., data structure, format, and terminology) and methodological (e.g., purpose of database, missing data, biases, and confounders) challenges, certain operational challenges (e.g., feasibility, governance, and multiple country involvement) can be unique to each region. As Schneeweiss12 remarks, it is important to separate the useful from the misleading. That is, it is critical to determine when and how RWD analysis may lead to valid evidence generation vs. when it will not. This may be achieved by developing use cases that address different questions (e.g., natural history of a disease, product utilization patterns and adherence to treatment, comparative effectiveness, and comparative safety) that can be interrogated using RWD to generate evidence. One such effort is aimed at evaluating the utility of RWD by reproducing 30 RCTs with observational RWD analyses to understand the circumstances in which observational studies can replicate the results of clinical trials.4 Gatto et al.13 describe a framework to generate valid and transparent RWE. The Structured Pre- and post-Approval Comparative study design framework to generate valid and transparent real-world Evidence (SPACE) was a process developed for identifying design elements, feasibility assessment, and validity considerations, as well as for documenting decisions. This framework starts with an articulated research question, which identifies the key components of the RCT (and pragmatic choices when required) needed to maximize validity. A goal of this framework is to improve dialogue and build trust among the different stakeholders. Digital technologies are touted to be a promising tool that can make health care more efficient and patient focused, while improving efficiency of drug development.14 Increasingly, digital devices, such as ingestible event markers, smart phones, and other wearables, are being used in clinical trials to enable better trial design and data collection.15 One such example is the "My Studies App" that was jointly developed by the FDA and private sector partners and is designed to facilitate the direct input of RWD by patients, which can be linked to electronic health data supporting traditional clinical trials, pragmatic trials, observational studies, and registries.16 Dickmann and Kothare17 postulate that digitally enabled RWD can help generate evidence for decision making in drug/device development, regulatory review, and payer coverage. They make a case for taking advantage of emerging and evolving digital health technologies to capture and aggregate real-world clinical data to support patient monitoring, personalized treatment, and research. However, as Shaywitz18 points out, a technology rarely arrives on the scene fully formed, and validation is needed to make it fit for purpose. These technologies will need to undergo analytical and clinical validation before being used for regulatory decision making. Novel approaches that utilize innovative technologies to incorporate patient-centric trial design and data collection methods can improve the efficiency of clinical trials. For example, pragmatic clinical trials can bridge the gap between research and care by taking into account patients' lifestyle. One such example is the Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-Term Effectiveness (ADAPTABLE; NCT02697916) trial run by the National Patient-Centered Clinical Research Network (PCORnet), which is integrated into routine clinical care and has minimal inclusion/exclusion criteria with no required treatment protocol beyond the assignment to one of the two doses of aspirin. EHRs and claims data capture primary and secondary end points as well as other relevant patient-reported outcomes.19 Another patient-centric trial design is the decentralized trial, which can be conducted in a real-world setting and can support both traditional clinical trial designs and other real-world study designs. Khozin and Coravos20 highlight that decentralized trials can reduce the burden of participation by facilitating in-home and remote monitoring. Effective use of these methods may extend the reach of clinical research to patients who are traditionally excluded in clinical trials (e.g., patients with comorbidities, elderly, or living in remote locations) by bringing the clinical trial to the patient. Engaging patients while designing and implementing real-world studies is beneficial for all stakeholders involved. As Nowell21 and Okun22 point out, patients are in the best position to provide real-world input on the performance of a therapeutic product. Beyond just safety and effectiveness, patients and caregivers can provide feedback on patient satisfaction and quality of life. Apart from patient-reported outcomes, other patient data (e.g., genomics, social media, and wearables) can be informative sources of RWD. However, these RWD need to be collected in a systematic and standardized way such that they become high-quality data and, consequently, a substantial source of RWE.21 The RWE that is generated needs to be translated into actions that can improve the everyday life of patients.22 As Shaywitz18 notes, this also calls for patient-oriented investigators who have a patient-focused vision and are willing to put the patients at the center of this ongoing conversation. Just like how RCTs cannot answer all the questions, RWD cannot address all the questions either. A learning healthcare system is one in which synergies between RCTs and RWD can be identified, thus leading to better patient care. Lee et al.23 describe the Applied Proteogenomics OrganizationaL Learning and Outcomes (APOLLO) network within the Department of Defense and Department of Veterans Affairs healthcare environment, which collects longitudinal RWD as part of routine health and cancer care delivery that can be used to generate RWE. This, in turn, enables a learning healthcare system. RWD generated from this endeavor would not only benefit veterans, active service members, and their families, but also benefit the community by releasing RWD (e.g., aggregate patient molecular profile data and deidentified clinical data) and RWE (e.g., clinical practice guidelines) to the public. Public distribution of the information and knowledge can, in turn, feed into future clinical, research, and regulatory decision making. From a global health viewpoint, Barrett and Heaton24 make a case that considerations should also be given to globally underserved populations. That is, RWE should benefit patients across the world (including low-/middle-income countries) and not just the developed world (high-income countries with advanced technological infrastructure). However, this is not that straightforward, as many challenges exist. Any bridging exercise needs to consider the local regulatory environment, infrastructure to support routine clinical care, understanding of disease epidemiology and progression, and advanced technological infrastructure needed to collect RWD (EHRs, claims databases, registries, etc.). Given the unmet need and the substantial potential benefits, precompetitive efforts can help advance the utility of RWD in a global public health setting. Multiple stakeholders, including biopharmaceutical industry, academia, patients and patient advocacy groups, rare disease groups, regulatory agencies, health systems, payers, and other stakeholders, have an interest in translating RWD to generate RWE. Although labor-intensive to set up and run, precompetitive collaborations among multiple stakeholders can aid in promoting alignment. For example, the Clinical Trial Transformation Initiative (CTTI), a public-private partnership that is focused on increasing quality and efficiency of clinical trials, is developing best approaches for using RWE generated from EHRs and claims data in RCTs as well as developing template consent language for RWD-sourced research.25 Another example of a precompetitive effort is the collaboration among the FDA, Friends of Cancer Research, National Cancer Institute, and other stakeholders that is focused on harmonizing reference standards for assessing tumor mutational burden to help identify patients who are likely to respond to immunotherapy.14 Additionally, many public meetings, including those hosted by Duke Margolis Center for Health Policy and National Academy of Sciences, Engineering, and Medicine, have served as a forum to discuss challenges and opportunities in collecting RWD and generating RWE. However, attention should be paid on education and training, as different stakeholders invested in this rapidly evolving space have varying levels of understanding of RWD and RWE. Rivera et al.26 discuss the need to embrace collaborative interdisciplinary teams and to enhance clinical data scientist training opportunities (Table 1 of ref. 26) to generate functionable knowledge that can be translated to actionable evidence. For traditionally underserved patient subgroups, harnessing real-world big data can help inform patient care. As Van Driest and Choi27 discuss, for pediatric patients, there is an opportunity to upcycle RWD (including pharmacokinetic and pharmacodynamic information) collected during routine clinical care to generate evidence to support treatment optimization in the postmarket setting as well as to accelerate efficient drug development for children. Similarly, for patients with cancers with rare driver mutations, Samant et al.28 articulate how thoughtful application of quantitative and qualitative research methods using clinico-genomic RWD can inform development of precision medicines for a small cohort. The ultimate goal is that we can leverage learnings from the case studies in oncology to accelerate drug development and personalize clinical care for patients with more common chronic diseases. After a drug has been approved, its dose and dosing regimen may become optimized in clinical practice and thus, may differ from the labeling recommendations. However, there has been limited validation and back translation of this clinical pharmacology RWD into therapeutic product labeling. Liu et al.29 surmise the opportunities and challenges in utilizing clinical pharmacology RWD to promote therapeutic individualization. This can be in the form of optimization of dose and dosing regimen, evaluation of benefit/risk in specific populations (in whom there is no or limited data), endpoint and biomarker development, and optimization of treatment for intrinsic (e.g., organ function) and extrinsic (e.g., drug interactions because of comedications) factors. For example, with drug-drug interactions, unanticipated, unrecognized, or mismanaged drug interactions are an important cause of morbidity and mortality. However, during investigational product development, it is impractical to evaluate every drug combination in clinical trials. Quinney30 highlights the advantages and disadvantages of different RWD sources to predict potential drug-drug interactions. However, at the moment, these bioinformatic assessments are hypothesis generating, although mechanistic understanding of drug action can improve confidence in these predictions. RWD can also help inform how labeling updates are being adopted by the community. For example, initiatives such as Information Exchange and Data Transformation (INFORMED) are exploring how labeling changes of approved products (nivolumab and pembrolizumab) from a weight-based dosing to flat dosing is being adopted by the community.14 Along with the opportunities for converting clinical pharmacology RWD to generate RWE are also the challenges. For example, inappropriately applied data and analysis may lead to flawed conclusions; therefore, it is essential for data scientists (including clinical pharmacologists and translational scientists) to develop methodologies based on sound epidemiologic principles. This will lead to effective extraction and application of critical information from RWD leading to sound evidence-based decisions. There is a great expectation for the utility of RWD to generate RWE, and it is up to all of us collectively to deliver on the promise of RWD and RWE. The authors wish to thank Daphne Guinn, Qi Liu, Jeffry Florian, Michael Pacanowski, Khair ElZarrad, Rajanikanth Madabushi, and Issam Zineh for providing thoughtful comments. No funding was received for this work. The authors declared no competing interests for this work. The contents of this article reflects the views of the authors and should not be construed to represent the US Food and Drug Administration's (FDA's) views or policies. No official support or endorsement by the FDA is intended or should be inferred. The mention of commercial products, their sources, or their use in connection with material reported herein is not to be construed as either an actual or implied endorsement of such products by the FDA.