Appraisal of Evidence and Control of Bias in the Kidney Disease Outcomes Quality Initiative Guideline Development Process

It is not evidence alone but a combination of evidence and judgment that drives most medical decision making. Limitations afflict even high-quality evidence; subjectivity influences even the most objective experts. Therefore, in the absence of successful clinical practice guidelines, prevalent practice suffers from an overreliance on flawed information and a susceptibility to uncontrolled bias. Ascribing causality to observational results (1), assuming the equivalence of surrogate and direct patient outcomes (2), and pursuing practices or interventions despite a dearth of evidence for safety (3) are signs that clinical practice may be operating without critical appraisal of the evidence or sufficient restraint of subjectivity. Practice under such conditions provides fertile ground for the seeds of influence, whatever their source. To be successful, a clinical practice guideline development process must deal effectively with both the limitations of available evidence and the inevitability of expert subjectivity. The required principles are straightforward: Tease out the strengths and the limitations of evidence, and control potential bias by adhering to scientific and methodologic rigor; foster an interdisciplinary and independent working group; and use an open, structured, and systematic development process. These were the founding principles for the National Kidney Foundation Kidney Disease Outcomes Quality Initiative (NKF-KDOQI) (4,5); they have helped to make KDOQI the pride of the nephrology community and the envy of those who develop non-nephrology guidelines, and they served as the organizational framework for formulating the 2006 KDOQI Clinical Practice Guidelines and Clinical Practice Recommendations for Anemia in Chronic Kidney Disease (6). Adhering to these principles serves as the strongest available antidote against potentially negative influence on clinical practice in our community, whether the source of the influence is industry, incomplete information, or unexamined belief. Adhering to scientific and methodologic rigor, for example, requires recognition that the source of the evidence affects its quality. Specifically, evidence that is of sufficiently high quality to establish the net benefit or net harm of a therapeutic intervention derives almost exclusively from studies of relevant clinical outcomes, including adverse events, in well-defined study populations using well-designed, carefully conducted, randomized, controlled trials (RCT). Trials that examine surrogate end points yield evidence of lower quality because the causal relationship among the intervention, surrogate end points, and clinical outcomes often is uncertain. In the worst case, interventions that improve surrogate end points may worsen health and shorten survival (2). A number of limitations can affect the evidence from both RCT and other study designs. The more narrowly defined the study hypothesis, the more tightly controlled the intervention; and the more highly selected the study population, the greater the difficulty in generalizing and applying study results to the practice setting, where the treatment options are varied, the patients are unselected, and what happens to patients (hard clinical outcomes) matters most. Evidence from observational studies and uncontrolled interventional trials usually is unsuitable for treatment guidelines except when it identifies harm, specifically adverse events that may have been too rare or too delayed to be observed in RCT. It is precisely when the clinical questions are urgent yet the limitations of evidence are substantial that experts are most needed. Measures to minimize potential expert bias are integral to the KDOQI evidence appraisal process. KDOQI anemia experts, as chairs and members of the work group, were chosen carefully on the basis of their expertise in the area of anemia and chronic kidney disease and the perspectives—both multidisciplinary and international—that each brings to the guideline development process. They were volunteers who served without compensation. All KDOQI volunteers were required to complete detailed conflict-of-interest statements, which were reviewed by the NKF’s scientific leadership, the work group chairs, and the KDOQI guideline development team, and were reproduced in each published guideline. To ensure independence of the work group, all work group members signed confidentiality statements at the onset of the guideline process, agreeing not to divulge drafts or discussion outside the work group at any time during the process. Industry sponsors were instructed to avoid any direct communication with work group members on the subject of the guidelines while they were being developed. Although other NKF volunteers assisted in raising funds to support guideline development, no one who participated in fundraising was permitted to serve on a work group. To use its collective expertise to maximum advantage, the work group used a protocol for systematic literature review (7), a structured approach to appraising the quality of evidence and grading the strength of the resulting recommendations (8), and a two-stage peer-review process (9). Work group members began the 2-yr guideline development process by defining the key clinical questions to be addressed and specifying, a priori, the information that needed to be culled from the literature. Aided by a 10-member methods team with expertise in systematic evidence review and guideline development and using predefined eligibility criteria for each topic, (6, Appendix Table 44, page S210) the work group screened 10,000 citations and identified 83 full-text articles for systematic data extraction. Data extraction of full-text articles, a process that demanded 6 mo of excruciatingly detailed work on the part of work group members, was the first step in extracting the evidence on which guideline statements ultimately would rest. The second step was to tabulate systematically in summary evidence tables for each target population (chronic kidney disease, hemodialysis, and peritoneal dialysis) the key trial elements and findings for each intervention or clinical practice. In the third step, evidence from the summary tables was compiled into a final table, an evidence profile, in which the evidence for each outcome, whether beneficial or harmful, was graded for quality and the importance of the outcome was weighted. Only after each outcome had been considered individually and all outcomes had been considered together was it possible then to derive a judgment of net benefit of the intervention and to affix to that judgment an evidence quality grade. Finally, the work group formulated a guideline statement that explicitly linked net benefit and quality of evidence. Two conditions had to be met for the work group to issue a guideline recommendation on an intervention or clinical practice: The overall quality of evidence had to be deemed high or at least moderately high, and there was a judgment of net benefit supporting a positive recommendation (e.g., CPG 2.1.1 Hemoglobin Lower Limit) or net harm supporting a negative recommendation (e.g., CPG 3.3.3 Use of Androgens). In the absence of high or moderately high quality for the overall body of evidence, the evidence was deemed to be insufficient to support a guideline recommendation, and a clinical practice recommendation (CPR) was issued (e.g., 2.1.2 Upper Limit of Hemoglobin, 3.2.4 Upper Level of Ferritin, 3.3.1 Use of Carnitine, 3.3.2 Use of Ascorbate). Exposing gaps in evidence contributes greatly to progress in both research and clinical practice. Identifying the absence of evidence or serious limitations in evidence unmasks unfounded assumptions, dispels prevailing belief systems, raises important safety concerns in patient care, encourages appropriate caution, discourages attempts to set clinical performance measures that lack foundation, removes impediments to needed research, and stimulates inquiry. Thus, the work group was unapologetic when offering a CPR that included the phrase “there is insufficient evidence to recommend.” Subjectivity cannot be excluded entirely from the assessment of limitations in the evidence, importance of individual outcomes, net benefit, or quality of evidence. Accordingly, the work group left a clear trail of its deliberations in the free-text rationale statements that accompany each guideline and the three-part evidence table sequence. The critical reader can follow in a stepwise manner the evidence that was available to the work group, the importance that the work group assigned to each outcome, and how the work group assessed quality of evidence. Full reproduction of each step of the evidence appraisal sequence adds a critical component to maintaining transparency of the work group process. The task of crafting guideline statements drew heavily on the multidisciplinary expertise and the breadth of experience of work group members. At this step, the object was to render guidelines and practice recommendations that both reflected the evidence accurately and met the particular needs of practitioners. To ensure that the process was open and inclusive, documents that contained each statement and accompanying rationale were maintained and edited using an online document library that was accessible continuously to work group members. Thus, the contributions and comments of each member could be captured and tracked as the document evolved. Through Web conferencing, conference calls, and work group meetings, statements were presented, revised, and honed until every word of every statement had gained the support of every work group member. A two-stage peer review further contributed to the rigor and the transparency of the guideline development process. In the first stage, the KDOQI Advisory Board, which consisted of more than 40 leaders from the nephrology community, provided an invaluable, internal peer review of an early draft. In the second stage, external peer review was conducted during a 2-mo open evaluation and comment period, when an advanced draft of the manuscript was made available for public review. More than one thousand external reviewers requested and were given access to the document; 126 returned comments and suggestions. The work group made final changes to the guideline statements only after public review was completed. Scientific rigor, interdisciplinary and independent work groups, structured and systematic evidence appraisal, and a two-stage peer review, of course, have steered the development process since formulation of the earliest DOQI guidelines. With continued evolution, the process has gained strength. The NKF has been instrumental in providing the influence, forum, organizational structure, and formidable resources needed to sustain the trajectory of improvement through maintaining a KDOQI advisory board, recruiting high-profile work group chairs and members, supporting a standing methods staff, and organizing the internal and external reviews. True to its mandate to improve the health of all patients with kidney disease, the NKF has broadened its perspectives to include partnering with the worldwide guideline development initiative Kidney Disease: Improving Global Outcomes (KDIGO) (10,11). Ten years have passed since the first Anemia Work Group, under the leadership of Joseph Eschbach, issued the original DOQI Anemia Guidelines that contributed so directly to a major and sustained improvement in the care of patients with dialysis-associated anemia (12,13). Ten years from now, we ardently hope that the 2006 KDOQI Anemia Guidelines will have stimulated sufficient new research to have prompted a second comprehensive revision. Perhaps KDIGO will replace KDOQI in the title, and the transition to a truly global guideline will be completed. Undoubtedly guideline statements will change. We trust, however, that by whatever title, scope, or wording, those new guidelines will continue to embody the founding principles of the NKF-sponsored guideline development process. Disclosures D.V.W. receives lecture and consultant fees from Amgen, American Regent, Ortho-Biotech, and Affymax, is a part-time employee of DaVita, and is co-chair of the KDOQI Anemia Work Group. K.-U.E. receives lecture and consultant fees from Amgen, Roche, Johnson & Johnson, and Affymax and is co-chair of the KDOQI Anemia Work Group. K.U. is a co-director of the KDOQI evidence review team. M.R. is vice-chair of the KDOQI Steering Committee and receives lecture and consulting fees from Amgen, NxStage, and DaVita. A.L. receives lecture and consultant fees and grant support from Amgen, Roche, Ortho-Biotech/Janssen-Cilag and is chair of the KDOQI Steering Committee.