Pitfalls of Randomized Controlled Trials in Stroke: How Can We Do Better?

HomeStroke: Vascular and Interventional NeurologyVol. 3, No. 4Pitfalls of Randomized Controlled Trials in Stroke: How Can We Do Better? Open AccessEditorialPDF/EPUBAboutView PDFView EPUBSections ToolsAdd to favoritesDownload citationsTrack citations ShareShare onFacebookTwitterLinked InMendeleyReddit Jump toOpen AccessEditorialPDF/EPUBPitfalls of Randomized Controlled Trials in Stroke: How Can We Do Better? Shadi Yaghi, James E. Siegler and Thanh N. Nguyen Shadi YaghiShadi Yaghi *Correspondence to: Shadi Yaghi, MD, FAHA, Department of Neurology, The Warren Alpert Medical School of Brown University, Eddy Street, Providence, RI 02903. E‐mail: E-mail Address: [email protected] https://orcid.org/0000-0003-0031-1004 , Department of Neurology, , The Warren Alpert Medical School of Brown University, , Providence, , RI, , The Neurovascular Research Center, , Lifespan, , Providence, , RI, Search for more papers by this author , James E. SieglerJames E. Siegler , Cooper Neurological Institute, , Cooper University Medical Center, , Camden, , NJ, , Cooper Medical School of Rowan University, , Camden, , NJ, Search for more papers by this author and Thanh N. NguyenThanh N. Nguyen , Department of Neurology, Boston Medical Center, , Boston University Chobanian and Avedisian School of Medicine, , Boston, , MA, Search for more papers by this author Originally published4 Jul 2023https://doi.org/10.1161/SVIN.123.000807Stroke: Vascular and Interventional Neurology. 2023;3:e000807Randomized controlled trials (RCTs) are considered to provide the highest level of evidence that drives practice changes in clinical medicine. The “magic of randomization” ensures balance of all potential measurable and unmeasurable confounders between exposure groups1 and blinding in many RCTs adds rigor and reduces the risk of selection bias.2 Many preconceived notions of therapies thought to help patients have been shown to either not help or harm patients, as in vitamin K antagonists for noncardioembolic stroke.3 However, results of RCTs in general and specifically in the stroke realm are imperfect and should be interpreted with caution for several reasons (Figure 1A).Download figureDownload PowerPointFigure 1. Steps in Randomized Controlled Trials and Pragmatic Trials. A, Upper part. Challenges of randomized controlled trial design along each step. These include enrollment bias, inadequate power, and limitations due to cost limiting generalizability to a real‐world setting. B, Lower part. Pragmatic randomized controlled trial (RCT) flow chart which includes rigorous observational design or randomization based on clinical equipoise to be incorporated into routine clinical practice and performed by site or alternate day enrollment, which involves blinded outcome adjudication. This leads to more generalizable results, lower cost, and potentially a larger sample with more robust statistical power.First, the time from a researcher proposing a study to enrolling the first patient may be several years. During this lengthy process, certain hypotheses, treatments or device technologies that were relevant at one time point may not be as relevant when the trial begins which may impact enrollment and the findings of such trials. For instance, the availability of left atrial occlusion as a treatment option for stroke prevention in atrial fibrillation can potentially impact ongoing anticoagulation resumption after intracerebral hemorrhage trials. Furthermore, advances in medical treatment for stroke prevention have led researchers to retest the safety and efficacy of carotid revascularization in asymptomatic patients, despite a prior trial showing benefit.4Second, the power calculations and sample size in RCTs are often based on unpublished preliminary or published data from observational studies without central adjudication of outcomes and thus the sample size calculations may be erroneously altering their findings and conclusions. In addition, feasibility and funding lines may impact the sample size or length of follow‐up that may be needed to show a statistically significant difference between the arms of RCTs.Third, many RCTs are designed to be pragmatic, or overly inclusive, and their results may not necessarily be applicable to the individualized medicine world or may be too selective to apply to the general population. The inclusion/exclusion criteria do not always consider minimizing harm and maximizing benefit based on their respective mechanisms when studying an intervention for a certain disease process. One size does not fit all. For example, early trials recruiting cryptogenic stroke patients with patent foramen ovale found no benefit with closure over medical management. Only with more selective criteria was there was a reduction in recurrent stroke risk with closure.5 That said, more inclusive trial designs, or tiered trials, may reveal clinically important associations that drive subsequent studies. For example, trials testing direct oral anticoagulants versus aspirin in patients with embolic stroke of undetermined source6, 7 were negative. However certain subgroups of embolic stroke of undetermined source were shown to benefit from direct oral anticoagulants in post‐hoc exploratory analyses.8 In addition, understanding mechanisms and risk factors for bleeding in embolic stroke of undetermined source patients can help exclude such patients from these trials or perhaps modify this risk.Fourth, RCTs may be subject to bias on several levels. It starts with the RCT design often geared toward the ability to screen, consent, and recruit which may potentially bias the RCT sample. Enrollment bias and lack of consecutive enrollment are major concerns. For instance, stroke prevention trials may be biased toward excluding those with moderate to severe strokes or preexisting disability9 due to the inability to consent the patient or being considered “too sick to enroll”.10 In addition, while clinical equipoise is a prerequisite for a site participating in a clinical trial, not all inclusion criteria may be considered as such and thus certain patients who qualify based on inclusion or exclusion criteria may be excluded by study sites. For example, “cherry picking” based on patient age in endovascular trials11 may have altered the results of these trials in both directions12, 13 (i.e., trials being negative due to treating younger patients outside the trials or trials being positive because of not enrolling older patients in the trials). Selective recruitment of patients most likely to benefit from the trial exposure, and most likely to be harmed without it, can falsely augment a treatment effect that is thought to apply to patients meeting broader inclusion criteria. Furthermore, sex,14 race, and ethnic15 differences exist in clinical trial enrollment, potentially limiting the generalizability of the findings to other populations. In addition, patients who consent to participate in a clinical trial may be more likely to follow up and comply with other stroke prevention measures which may potentially lead to a lower event rate in clinical trials as compared with real world studies. Certain participating sites may have competing studies with overlapping inclusion criteria and if approaching patients for one trial versus another is not a random process, this can bias the population enrolled in an RCT. The close monitoring, additional resources, and aggressive medical management of patients enrolled into trials may not be replicable or available in clinical practice.All these factors may have influenced discrepancies between RCTs and real‐world studies. For example, the recurrent stroke rate in an RCT of medically treated patients with symptomatic intracranial stenosis was nearly 5% at 30 days16 which was much lower than the corresponding rate reported in real world study (22%)17 leading to the question whether the findings from this trial were generalizable to a real‐world population.Fifth, while investigator and patient blinding are often integral to RCTs, this is not always the case. Even when appropriate measures are taken, accidental unblinding may occur and potentially impact outcome assessments. For example, certain treatments may cause side effects or alter laboratory values making it easier for the patient or a “blinded” adjudicator to know which arm the patient is in. On the other hand, the “placebo effect” may alter findings in trials of patient‐reported outcomes.18Sixth, well powered RCTs comparing 2 treatments in uncommon diseases such as cervical artery dissection and cerebral venous thrombosis and in other disease with low event rates may be a challenge.19, 20 Large RCTs for such conditions will require a very large number of sites with high recruitment rates and large budgets and thus may not be feasible.Seventh, not all trials–even if neutral or positive–will lead to change in practice.21 This could be related to cost of the novel management strategy, potential loss of equipoise in favor of or against an intervention, not accounting for an individualized treatment approach, lack of feasibility in clinical practice, and issues related to trial's primary outcome. Moreover, the trial should take into consideration a primary outcome that is clinically meaningful to the patient and designed in the patient's best interest.22Finally, there are drawbacks in the government funding process for RCTs, which includes a lengthy process and limitation of funding. These aspects could be mitigated in industry funded trials.Thus, improvement in clinical trial methodology and dismantling some of the barriers to its execution are needed to generate the best quality evidence to impact practice (Table 1). The high cost and administrative burden in a trial may be barriers prohibitive to the execution of the trial. When funding an RCT, funding agencies should consider where the field is heading and how changes can potentially impact recruitment and applicability of the findings from a proposed RCT. An explanatory model with primary and secondary hypothesis of the target question may bring pragmatism to clinical trials and allow several questions to be tested in parallel within 1 trial.23 On the other hand, the inclusion and exclusion criteria should focus on an individualized risk/benefit assessment minimizing the risk from the exposure and ensuring that the hypothesized benefit outweighs this risk. In addition, RCTs should take into account the effect size of a certain treatment as opposed to only being focused on achieving statistical significance.24 Moreover, RCTs should strongly consider coupling with prospective observational studies including patients from the same sites who were not enrolled and study the treatment effect using inverse probability weighting or other means of propensity score matching to verify generalizability of RCT findings with prospective real world observational data. It is also crucial that RCT funding covers the collection of characteristics of patients eligible but not enrolled in the RCT to ensure there is no recruitment bias. Another key step is to ensure appropriate blinding measures are in place and identifying and mitigating ways outcome adjudicators can be unblinded and eliminating access to any potential information that could cause unblinding and bias the results. In addition, for rare diseases, it is reasonable to rely on well‐designed comparative effectiveness observational studies or meta‐analyses of small RCTs to help generate evidence. Finally, it is crucial to ensure that clinical trials if positive or negative, findings can be easily translated into clinical practice.Table 1. Dilemmas and Potential Solutions in Randomized Trial Design and InterpretationLimitationPossible solutionInclusion/exclusion criteria over‐ or under‐selecting trial candidatesConsider tiered strategy or prespecified secondary analyses of more inclusive randomized trialsSmall target populationMeta analyze trial result data with data from published observational cohort studiesFunding limitationsEnsure the most relevant clinical data are collected for trial participants. Consider sample size estimations based on effect size rather than simply statistical significanceRecruitment biasConsider collecting data for patients eligible but not enrolled and for patients ineligible for inclusionUnintentional unblindingEnsure measures are taken to prevent investigators or patients from knowing their exposure group (eg, censored laboratory or imaging data, sham procedures)Clinical trial results do not translate into practiceEnsure outcomes are clinically important to providers and outcomes. Survey patients and health care providers prior to conducting the studies of whether such study findings will translate into practiceProlonged review process and funding limitations in government funded trialsConsider applying for industry fundingJohn Wiley & Sons, Ltd.When interpreting the findings of RCTs, many questions arise. Do the RCT results apply to this time and place? Did the RCT attempt a pragmatic or individualized medicine approach? Did the study sites aim and attempt to recruit every eligible patient or was there recruitment bias? How many patients eligible were not enrolled and do their characteristics differ from those enrolled? What is the treatment effect size and was the trial powered to answer the question? Is the primary outcome in the trial clinically important and will the study be easily translated into clinical practice?Due to the above limitations and since a RCT sample may not always reflect the “real world,” should we move toward utilizing “pragmatic care randomized studies”?25 In these trials, randomization is integrated into the clinical practice and with blinding, independent outcome assessments, and frailty adjustment, this may provide a rigorous way to answer many of the uncertainties in clinical medicine, incurring a lower cost to answer research questions. In addition, this can allow ease in translating findings into clinical practice as study interventions are studied in real world populations (Figure 1B). For example, in a pragmatic care RCT, the study intervention meeting criteria to proceed to a clinical trial setting is tested in a real‐world setting. For instance, a pragmatic care randomized study testing a treatment would randomize sites to 1 treatment or the other. This way, the representative sample is a real‐world sample and results can be generalized to the real‐world population.In summary, further steps are necessary to improve the quality of RCT methodology. Rigorous alternatives such as pragmatic care randomized studies should be considered. Such an approach could lead to improvements in study design, higher patient recruitment, lower cost, and greater generalizability.Sources of FundingNone.DisclosuresT. Nguyen reports advisory board with Idorsia (not related); research support from Medtronic, SVIN (2021).AcknowledgmentsNone.Footnotes*Correspondence to: Shadi Yaghi, MD, FAHA, Department of Neurology, The Warren Alpert Medical School of Brown University, Eddy Street, Providence, RI 02903. E‐mail: [email protected]comThe opinions expressed in this article are not necessarily those of the editors, the American Heart Association, or the Society of Vascular and Interventional Neurology.References1 Collins R, Bowman L, Landray M, Peto R. The magic of randomization versus the myth of real‐world evidence. N Engl J Med. 2020; 382:674‐678.CrossrefMedlineGoogle Scholar2 Boutron I, Estellat C, Guittet L, Dechartres A, Sackett DL, Hróbjartsson A, Ravaud P. Methods of blinding in reports of randomized controlled trials assessing pharmacologic treatments: a systematic review. PLoS Med. 2006; 3:e425.CrossrefMedlineGoogle Scholar3 Mohr JP, Thompson JL, Lazar RM, Levin B, Sacco RL, Furie KL, Kistler JP, Albers GW, Pettigrew LC, Adams HP, et al. A comparison of warfarin and aspirin for the prevention of recurrent ischemic stroke. N Engl J Med. 2001; 345:1444‐1451.CrossrefMedlineGoogle Scholar4 Endarterectomy for asymptomatic carotid artery stenosis. Executive committee for the asymptomatic carotid atherosclerosis study. JAMA. 1995; 273:1421‐1428.CrossrefMedlineGoogle Scholar5 Favilla CG, Messé SR. New data support patent foramen ovale closure after stroke. Stroke. 2018; 49:262‐264.LinkGoogle Scholar6 Hart RG, Sharma M, Mundl H, Kasner SE, Bangdiwala SI, Berkowitz SD, Pare G, Kirsch B, Pogue J, Pater C, et al. Rivaroxaban for stroke prevention after embolic stroke of undetermined source. N Engl J Med. 2018; 378:2191‐2201.CrossrefMedlineGoogle Scholar7 Diener HC, Sacco RL, Easton JD, Granger CB, Bernstein RA, Uchiyama S, Kreuzer J, Cronin L, Cotton D, Grauer C, et al. Dabigatran for prevention of stroke after embolic stroke of undetermined source. N Engl J Med. 2019; 380:1906‐1917.CrossrefMedlineGoogle Scholar8 Healey JS, Gladstone DJ, Swaminathan B, Eckstein J, Mundl H, Epstein AE, Haeusler KG, Mikulik R, Kasner SE, Toni D, et al. Recurrent stroke with rivaroxaban compared with aspirin according to predictors of atrial fibrillation: secondary analysis of the navigate esus randomized clinical trial. JAMA Neurol. 2019; 76:764‐773.CrossrefMedlineGoogle Scholar9 Siegler JE, Qureshi MM, Nogueira RG, Tanaka K, Nagel S, Michel P, Vigilante N, Ribo M, Yamagami H, Yoshimura S, et al. Endovascular vs medical management for late anterior large vessel occlusion with prestroke disability: analysis of clear and rescue‐Japan. Neurology. 2023; 100:e751‐e763.CrossrefGoogle Scholar10 Kasner SE, Siegler JE, Zamzam A, Kleindorfer D. Expanding eligibility in stroke prevention trials to patients with early disability. J Stroke Cerebrovasc Dis. 2019; 28:2268‐2272.CrossrefGoogle Scholar11 Ospel JM, Kashani N, Almekhlafi M, Chapot R, Goyal M. Influence of age on evt treatment decision in patients with low aspects: results of a multinational survey and its implications. Clin Neuroradiol. 2020; 30:37‐40.CrossrefGoogle Scholar12 Fiehler J, Thomalla G, Bendszus M. Cherry‐picking the wrong patients?Clin Neuroradiol. 2020; 30:41‐42.CrossrefGoogle Scholar13 Zaidat OO, Liebeskind DS, Jadhav AP, Ortega‐Gutierrez S, Nguyen TN, Haussen DC, Yavagal DR, Froehler MT, Jahan R, Nogueira RG, et al. Impact of age and alberta stroke program early computed tomography score 0 to 5 on mechanical thrombectomy outcomes: analysis from the STRATIS registry. Stroke. 2021; 52:2220‐2228.LinkGoogle Scholar14 Strong B, Pudar J, Thrift AG, Howard VJ, Hussain M, Carcel C, de Los Campos G, Reeves MJ. Sex disparities in enrollment in recent randomized clinical trials of acute stroke: a meta‐analysis. JAMA Neurol. 2021; 78:666‐677.CrossrefMedlineGoogle Scholar15 Burke JF, Brown DL, Lisabeth LD, Sanchez BN, Morgenstern LB. Enrollment of women and minorities in NINDS trials. Neurology. 2011; 76:354‐360.CrossrefMedlineGoogle Scholar16 Chimowitz MI, Lynn MJ, Derdeyn CP, Turan TN, Fiorella D, Lane BF, Janis LS, Lutsep HL, Barnwell SL, Waters MF, et al. Stenting versus aggressive medical therapy for intracranial arterial stenosis. N Engl J Med. 2011; 365:993‐1003.CrossrefMedlineGoogle Scholar17 Sangha RS, Naidech AM, Corado C, Ansari SA, Prabhakaran S. Challenges in the medical management of symptomatic intracranial stenosis in an urban setting. Stroke. 2017; 48:2158‐2163.AbstractGoogle Scholar18 Brown WA. Expectation, the placebo effect and the response to treatment. R I Med J (2013). 2015; 98:19‐21.Google Scholar19 Kasner SE. Antithrombotic therapy for cervical arterial dissection. Lancet Neurol. 2021; 20:328‐329.CrossrefGoogle Scholar20 Yaghi S, Shu L, Bakradze E, Salehi Omran S, Giles JA, Amar JY, Henninger N, Elnazeir M, Liberman AL, Moncrieffe K, et al. Direct oral anticoagulants versus warfarin in the treatment of cerebral venous thrombosis (action‐CVT): a multicenter international study. Stroke. 2022; 53:728‐738.LinkGoogle Scholar21 Edwards C, Drumm B, Siegler JE, Schonewille WJ, Klein P, Huo X, Chen Y, Abdalkader M, Qureshi MM, Starbian D, et al. Basilar artery occlusion management: specialist perspectives from an international survey. J Neuroimaging. 2023.Google Scholar22 Darsaut TE, Collins J, Raymond J. Patients may be right: clinical research should be designed in their best medical interest. Neurochirurgie. 2023; 69:101391.Google Scholar23 Nguyen TN, Raymond J, Nogueira RG, Fischer U, Siegler JE. The problem of restrictive thrombectomy trial eligibility criteria. Stroke. 2022; 53:2988‐2990.LinkGoogle Scholar24 Livingston EH, Elliot A, Hynan L, Cao J. Effect size estimation: a necessary component of statistical analysis. Arch Surg. 2009; 144:706‐712.Google Scholar25 Darsaut TE, Raymond J. Ethical care requires pragmatic care research to guide medical practice under uncertainty. Trials. 2021; 22:143.Google Scholar eLetters(0)eLetters should relate to an article recently published in the journal and are not a forum for providing unpublished data. Comments are reviewed for appropriate use of tone and language. Comments are not peer-reviewed. Acceptable comments are posted to the journal website only. Comments are not published in an issue and are not indexed in PubMed. Comments should be no longer than 500 words and will only be posted online. References are limited to 10. Authors of the article cited in the comment will be invited to reply, as appropriate.Comments and feedback on AHA/ASA Scientific Statements and Guidelines should be directed to the AHA/ASA Manuscript Oversight Committee via its Correspondence page.Sign In to Submit a Response to This Article Previous Back to top Next FiguresReferencesRelatedDetails July 2023Vol 3, Issue 4 Article InformationMetrics © 2023 The Authors. Published on behalf of the American Heart Association, Inc., and the Society of Vascular and Interventional Neurology by Wiley Periodicals LLC.This is an open access article under the terms of the Creative Commons Attribution‐NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.https://doi.org/10.1161/SVIN.123.000807 Manuscript receivedMarch 13, 2023Manuscript acceptedApril 24, 2023Originally publishedJuly 4, 2023 Keywordslimitationseditorialsrandomized controlled trialpragmatic randomized controlled trialbiasPDF download