Sham or Scam

To the Editor: Timely and accurate clinical trials fulfill an important need in medicine. As health care professionals, we are responsible for determining the efficacy, and even the necessity, of new surgical procedures as they are introduced into the field. Nevertheless, very often a new procedure is not formally investigated in a clinical trial before it gains widespread support and becomes a part of “standard of care.” A number of historical examples come to mind, most notably, carotid endarterectomy, which was once widely prescribed for patients without the availability of high-level data to scientifically govern its utilization. After several Level-I studies published on the procedure defined its utility in only some sub-populations of patients,1–5 we were reminded that physician intuition alone does not adequately replace the conclusions gleaned from rigorous controlled clinical trials. Recently, 2 randomized controlled trials that studied vertebroplasty against a sham procedure, considered the ‘gold standard’ comparator when investigating effectiveness of a surgical procedure, were published in the New England Journal of Medicine.6,7 These studies have received a high degree of attention since both of them suggested that vertebroplasty provided no better outcomes than the sham control. During the last decade, vertebroplasty has been effectively established as an important part of the standard of care for patients having painful osteoporotic vertebral compression fractures. Consequently, the conclusions of these studies have had a profound impact on the medical community. Since their publication in August 2009, professional organizations have mobilized to provide varying position statements as a response to these 2 articles. Clinical investigations in which a sham control is utilized are few and far between since they are difficult to perform for a number of reasons including their high cost, the desire of patients to participate, and clinical feasibility. Especially difficult is performing such a trial on a procedure already excepted as a standard of care by patients and physicians alike. As the design of a clinical study becomes more complicated, increased diligence is required during its planning and implementation. This diligence must persist with the interpretation of data as well. When analyzing and reporting a clinical study, it is important to appreciate the ramifications of modifying key elements of the planned study protocol. Both of these highly publicized vertebroplasty articles demonstrated significant study design flaws that affect the integrity of their conclusions. Despite this, the investigators completing both of these studies must be commended for their work. These flaws may be attributed to decisions made while the study was ongoing where important study design parameters, including relaxing the study subject inclusion criteria, reducing the target sample size post hoc, and providing the opportunity for patients to self-select themselves into the study, were allowed to occur. These actions in effect result in the inadvertent manipulation of results. The overall effect of these manipulations was a lack of study power for detecting a significant difference between treatment groups. When planning a Level 1 clinical trial designed to evaluate a surgical procedure, selection of the most appropriate control is frequently bedeviled by ethical issues.8–20 Ethical conduct of a controlled clinical research trial demands ‘clinical equipoise’—there must be genuine uncertainty within the expert medical community as to what would be considered the preferred treatment.21 Should a sham procedure be offered to a patient if it is thought to have no therapeutic value? Would a sham be offered to a patient if it is shown to have therapeutic value equal to its active comparator? This is particularly controversial when the procedure for which it is being used as a comparator is considered efficacious. Over the last decade, numerous clinical studies have shown vertebroplasty to be acceptable to patients and to provide significant and meaningful pain relief to the majority of those treated.22–26 In the sham-controlled trials recently published in the New England Journal of Medicine, it is questionable whether clinical equipoise was achieved since the vertebroplasty procedure was already well-established as an important part of acceptable clinical care. Patient partiality toward pursuing a popular standard of care treatment complicates enrollment in a study where the accepted care is compared against a sham treatment. Accuracy of such a study's conclusions can be questioned since there is high likelihood that patients may self-select themselves into or out of the study, or may drop out completely, if they find their symptoms have not been alleviated with treatment. Both the Kallmes et al and Buchbinder et al studies may have been affected by significant upstream selection bias: 1682 of 1813 patients screened by Kallmes et al were excluded for various reasons; Buchbinder et al. enrolled 78 of 468 patients. Buchbinder et al conceded that selection bias could not be entirely ruled out, “since 30% of potentially eligible participants declined to participate in the study.” Patients with severe pain are more likely to shy away from sham studies for fear of being placed in the control group. This can leave a study group that is not representative of the patient population the intervention was designed to treat. Kallmes et al acknowledged that baseline medication use was not recorded, which would have provided a better indication of the severity of the patient population's pain level prior to undergoing surgery. If patients are less likely to benefit from the treatment, significant improvement is more difficult to show statistically. Compounding the effects associated with study subject self-selection, the sham surgery in both of these studies was undeniably an active treatment. In both cases, an incision was made and local anesthetic was injected into the pain affected region. The terms “sham” and “placebo” are often mistakenly used interchangeably. Though differentiating between sham procedures and placeboes may seem a semantic argument, it is an important distinction to make as the two terms carry with them very different connotations. When studies are controversial enough to attract national media attention, the connotative influence of a term like “placebo” becomes amplified. In Buchbinder et al study, the sham was referred to as a placebo treatment—but placeboes, as the term has historically been taken, are passive by nature. They have no intrinsic influence for good or ill; any effects caused in the patient would be generated by the patient's response to the perception of undergoing an active treatment. When sham treatments are not true placebo treatments, then the mindset of the clinical investigator must be adjusted, where the sham surgical procedure should be placed into the active treatment category and statistically evaluated as such. Logically, if sham surgeries are being argued in these studies as viable alternatives, then proving “non-inferiority” of the study treatment, rather than “superiority,” should be the goal. The sham surgery groups in both studies demonstrated strong positive results at one month. Patients in Buchbinder's study had reduced mean VAS pain scores from 7.1 to 5.4, a difference of 1.7 points, and Kallmes' patients mean VAS pain scores fell from 7.2 to 4.6, a difference of 2.6 points. This shows that the average difference for both sham groups would exceed even what most would consider a stringently-defined minimal clinically important difference for mean change in VAS pain score. Even if these reductions at 1 month were an anomaly, one would expect their effects to lessen with time. However, Buchbinder's sham surgery patients continued to improve with time, with their mean VAS scores continuing to decline to a difference of 2.1 points. This trend makes one suspect that the patients included in these studies may not have required a surgical procedure. Would the patients included in these studies have been more suitably treated using more conservative care? Those running clinical trials often run into difficulties recruiting patients and the investigators overseeing these trials were no exception. Study sample size is inextricably tied to study power, where detection of a significant difference is directly related to study sample size, sampling variability, and the observed effect of the study treatment. In the NEJM studies, both sets of investigators decreased the target sample size during the trial. By reducing the sample size required by the a priori power calculation, it became necessary for both of these studies to demonstrate a larger treatment effect than planned a priori in order for statistical significance to be accepted. The minimal clinically important difference (MCID) is an important concept for showing significant improvement, especially for describing the strength of results on a per-patient basis. Each patient is characterized as either having met or failed to meet a defined improvement criterion. Kallmes revealed 64% of vertebroplasty patients and 48% of control patients achieved their accepted MCID of 1.5 points (P = .06). Allowing a larger study size as originally planned would have likely made this result significant with a P < .05 favoring the vertebroplasty arm. Buchbinder does not provide the MCID results, although she mentioned that “the observed differences between groups were smaller than the MCIDs for all outcomes ...” Failure to produce the results may indicate an author bias if results favored vertebroplasty. This potential for having observed self-selection bias is notable when the results of these 2 studies are compared to those reported by Eck et al who reviewed the literature and evaluated it using a meta-analysis-type design.27 The study included 103 vertebroplasty studies having a total of 7587 patients. Evaluating pain using a 10-point visual analogue scale (VAS) score, Eck reported a mean preoperative VAS score of 8.36, declining to 2.68 postoperatively (P < .001). In contrast, Kallmes and Buchbinder reported lower mean preoperative VAS scores (6.9 and 7.4, respectively) with substantially less reduction after the surgical procedures. The improvement in pain scores after vertebroplasty in Eck's meta-analysis was 67.9%, markedly higher than that obtained by Kallmes (43.5%) and Buchbinder (31.1%). This can be accounted for in part by the fact that a patient with higher baseline pain naturally has a greater potential for improvement than one with a more moderate VAS score for whom beneficial effects become less pronounced. The low initial recruitment in Kallmes et al study actually resulted in a liberal relaxation of pain criteria, so that patients with VAS ratings of as little as 3 on the 10-point scale were included in the trial. The high degree of attention that both of these randomized controlled vertebroplasty trials has received attests to the great deal of faith that we place in the outcomes of high-level studies. However, they are also excellent examples of the need for scrutiny in assessing study design and recruitment procedures for randomized controlled trials. Most importantly, they raise questions on the appropriateness of sham surgery as a control when a viable treatment is already an established clinical practice. They also demonstrate that great care must be taken when study design parameters are modified during an ongoing trial. We believe that medical research article authors and reviewers need to provide a fair and balanced view of the medical research that actually took place so that the reader can precisely assess which conclusions are warranted and which ones may be flawed. This means that even in cases where high-level research is suggested as having taken place, it is up to the professional medical community to ensure that the scientific evidence is critically assessed and interpreted appropriately. Bruce Frankel Charleston, South Carolina Daniel A. Bloch Stanford, California