Estimating the reproducibility of psychotherapy effects in mood and anxiety disorders: the possible utility of multicenter trials

Estimating the reproducibility of psychotherapy effects is essential. This is particularly crucial for trials with large effects, as the inclusion of false-positive trials can lead to erroneous conclusions about treatment efficacy in research syntheses1. Multicenter studies allow researchers to estimate the reproducibility of effects directly across centers under comparable study conditions (e.g., comparable enrollment procedures, inclusion/­exclusion criteria, assessment plans). In an important sense, implementation of trials at various centers is close to a direct replication of findings. Accordingly, recent standards recognize the benefit of describing individual center effects in multicenter studies2. We aimed to review what we know about center effects in multicenter trials with psychotherapy components for the treatment of mood and anxiety disorders. We examined the extent to which such multicenter trials: a) reported the variability of treatment outcomes for individual centers (i.e., random center effects) and/or b) provided an estimate of the strengths of treatment by center interactions (i.e., fixed center effects)3. To obtain a representative sample of recent multicenter studies, we conducted on July 18, 2020 a systematic search of studies indexed between 2010 and 2020 in Medline, PsycINFO and Educational Resources Information Center (ERIC). We used the key words "multicenter or multi-center" combined with "psychotherapy or therapy or counseling" and "depression or anxiety" and publication type "clinical trial" and "adult population". We identified 184 papers, of which 30 referred to treatment outcomes in a multicenter randomized clinical trial (overall 6,638 patients, range 22-1025). Descriptive characteristics of the 30 identified multicenter studies can be obtained from the authors upon request. In all 30 reports, "multicenter" was mentioned in the title or abstract and in the Methods section. The number of centers ranged from 2 to 30, but in four reports this number was not reported. The majority of the trials investigated treatment efficacy (e.g., changes in symptoms) and four studies investigated economic outcomes (e.g., cost-effectiveness analyses). In 20 studies, at least one significant treatment effect was reported (max. Cohen's d ranged from 0.23 to 3.44). Only one (3%) out of the 30 studies4 considered sites a random factor, thereby permitting conclusions about variability in outcomes due to sites in general. Only three (10%) studies5-7 reported an estimate of the treatment by center interactions. Furthermore, seven studies reported that center effects were "not significant", without further specification of the effect. Among the seven studies with large significant treatment contrasts (max. Cohen's d >0.80), only one4 reported a statistical estimate of a center effect. One of the strengths of multicenter studies is the opportunity to estimate the reproducibility of effects. The results of our systematic review indicate that, although studies state clearly that they involve multiple sites and often indicate that this adds to the importance of the trial, they typically do not use the full potential of this design to estimate center effects (either random or fixed), thereby obscuring evidence about reproducibility of effects. To properly assess the degree to which results are reproducible, we recommend that the authors of multicenter studies report the outcomes for all centers and estimate center effects (i.e., differences in effects amongst centers)8.