Local validation and calibration of pre‐eclampsia screening algorithms

A recent Practice Guideline issued by the International Society of Ultrasound in Obstetrics and Gynecology recommends a combination of a-priori factors and pregnancy-specific markers, determined at 11–13 weeks' gestation, as the most efficient method of identifying women at risk of pre-eclampsia (PE)1. Such screening followed by daily low-dose soluble aspirin treatment in those at high risk substantially reduces the incidence of the disorder2. This has been established using the highest standard of proof, namely a large international double-blind placebo-controlled randomized clinical trial, the Combined Multimarker Screening and Randomized Patient Treatment with Aspirin for Evidence-Based Preeclampsia Prevention trial (ASPRE)3, 4. A total of almost 27 000 women were screened and the 11% with the highest risk following screening were randomized to placebo or 150 mg of aspirin/night. The incidence of ‘preterm’ PE (requiring delivery before 37 weeks' gestation) was 4.3% in the control arm and 1.6% in the aspirin arm, a 62% reduction; for ‘very early’ PE (delivery before 32 weeks), the reduction was 78%. A secondary analysis of ASPRE found that, when treatment compliance was at least 90%, the reduction in the incidence of preterm PE was 76%5. The screening test in ASPRE used the Fetal Medicine Foundation (FMF) algorithm, developed at King's College Hospital (KCH), London, UK, to calculate the risk of PE. The algorithm is based on data accumulated over more than a decade, in a series of non-intervention studies in which women were tested but the results were not used clinically. A number of markers were considered and most attention was paid to uterine artery (UtA) Doppler pulsatility index (PI), mean arterial pressure (MAP), a weighted average of systolic and diastolic blood pressure, and maternal serum pregnancy-associated plasma protein-A (PAPP-A) and placental growth factor (PlGF). When the latest version of the FMF algorithm was applied to 292 cases of preterm PE and almost 35 000 unaffected pregnancies, screened at KCH and the nearby Medway Maritime Hospital, the observed detection rate was 75% at a false-positive rate of 10%; among 66 very early cases, the detection rate was 89% at a false-positive rate of 10%6. Although the FMF algorithm was based on a pregnant population largely from South-East England, it has universal applicability and will, theoretically, achieve high screening performance elsewhere. However, there are two provisos: (1) validated adherence to published protocols for UtA-PI and MAP measurement; and (2) calibration to fit local conditions, which can be done at any medical center using statistical methods that require no special expertise. These conditions were ensured prior to the start of the ASPRE trial in each of the 13 participating maternity hospitals in six countries (UK, Spain, Belgium, Greece, Italy and Israel). Results of the non-intervention phase have been published for 12 of the hospitals: there were 59 cases of preterm PE and 8536 unaffected pregnancies with a detection rate of 80% at a false-positive rate of 10%; all 17 very early PE cases were detected7. FMF risk estimation is based on a sophisticated model which takes into account the competing risks of PE and spontaneous early delivery unrelated to the disorder. Several a-priori risk factors are used to estimate the incidence of PE requiring delivery at each week of gestation. These factors are maternal age, weight and height, method of conception, parity, interpregnancy interval, previous PE, gestational age at delivery in the previous pregnancy, family history of PE, and maternal history of chronic hypertension, systemic lupus, antiphospholipid syndrome and diabetes mellitus. This component of the algorithm is based on 2704 cases of PE and more than 120 000 unaffected pregnancies8. Levels of each of the four biomarkers are adjusted for significant covariables, including gestational age and maternal weight, by expressing the values as multiples of the median (MoM). The equations used in these calculations were also obtained using data on large numbers of women screened at KCH. A multivariate Gaussian model is then used to calculate a series of likelihood ratios (LRs) derived from overlapping distributions of pregnancies with PE and unaffected pregnancies at each week of delivery. The model parameters are means, SDs and correlation coefficients of log MoM. For each marker, the distribution of MoM values in pregnancies with PE becomes closer to that in unaffected pregnancies as the gestational age at which delivery is required approaches term. Hence, the affected mean used in the Gaussian model is based on linear regression9. The LRs are applied to the a-priori risks and the results summed up to the gestational age at delivery of interest to obtain the final risk. As the FMF has done previously for other biophysical pregnancy markers, such as nuchal translucency and the nasal bone, great effort was made to standardize the measurement of UtA-PI and MAP10, 11. In the standardized protocol for UtA-PI measurement, a transabdominal probe is used to identify the UtAs and obtain a sagittal section of the cervix using color Doppler. The transducer is tilted from side to side to identify the UtAs at the level of the internal os. The sampling gate is set at 2 mm, covering the whole vessel, with an angle of insonation < 30° and peak systolic velocity ≥ 60 cm/s. UtA-PI is the average of the measurements of both arteries. For MAP, maternal systolic and diastolic blood pressure are measured using an FMF-certified automated blood-pressure device. In the standardized protocol, the woman is seated comfortably with her back straight and legs uncrossed; the arms are supported at heart level and are free of clothing; cuff size is selected depending on the mid-arm circumference; neither observer nor patient should talk during the measurement; blood pressure is measured in both arms simultaneously. Measurements are repeated after 1 min. MAP is the average of systolic and diastolic readings, calculated as the sum of diastolic + (systolic − diastolic)/3. It is highly recommended that each center undergoes training with FMF supervision before starting to screen women using these biophysical markers. Such training and external monitoring was carried out at seven National Health Service maternity hospitals in England for the Screening Program for pRE-Eclampsia (SPREE) project12. A total of 142 cases of preterm PE and more than 16 000 unaffected pregnancies were screened using the FMF algorithm, yielding a detection rate of 81% at a false-positive rate of 10%. Another prerequisite is access to risk-calculation software that uses a well-documented algorithm and can be calibrated to fit local conditions. Three aspects may need calibration: the normal median equations used for calculating MoMs for all four markers; the SD of log MoM values for each marker; and the risk cut-off. The preset equations and SD could result in inaccurate or imprecise calculation of LRs. This is illustrated in Figure 1 for PlGF, with substantial deviations from the preset values. The preset cut-off could result in a lower than expected detection rate or a higher false-positive rate. As is often found in Down-syndrome screening, preset normal median equations may not be suitable in a particular locality. If, after a period of training, in a consecutive series of at least 50 pregnancies, the observed median value for UtA-PI or MAP differs substantially from 1 MoM, retraining should be considered. In Australia, a quality-control scheme for sonographers measuring UtA-PI in PE screening considered an acceptable range to be 0.90–1.10 MoM13. The spread of MAP values is much smaller than that of UtA-PI and the equivalent in terms of Mahalanobis distance is 0.97–1.03 MoM. If the problem persists despite retraining, it would be advisable to multiply the preset equation by a ‘scale factor’ of the observed median MoM. For logarithmic equations, this is achieved by adding log median to the intercept. If the median value for PAPP-A or PlGF serum markers in the same series is outside the range 0.90–1.10 MoM, it would be best to simply replace the preset normal median equations. If one or both markers are already being used for Down-syndrome screening, the equations should be replaced by those in current use. If not, or if the new median is still out of range, new equations should be derived. Ideally, a consecutive series of at least 300 samples is needed. For each half week of gestation from 11 + 0 to 13 + 6 weeks, the median concentration and median number of days is obtained and a regression of medians carried out, weighted by the number in the half week. Alternatively, aliquots of the samples could be sent for retesting in a reference laboratory to derive a scale factor. After recalculating the MoMs, maternal weight should be divided into six groups and a regression of median MoM, weighted on median weight, performed. Equations of different shapes can be investigated to find the best fit. For each marker, the SD of log MoM should be estimated in a consecutive series of at least 300 pregnancies. If maternal serum PAPP-A or PlGF is already being measured for Down-syndrome screening, the series can include data collected previously. After any changes to the preset median equations, the 10th and 90th percentile MoMs should be derived and the SD estimated by dividing the difference between the logs of the respective MoMs by 2.563. Alternatively, it can be estimated from the slope of a Q–Q plot. These non-parametric approaches are more robust than is direct estimation of SD as they ameliorate the influence of outliers. The observed SD is compared to the preset value and, if they are similar, no further action is required. An observed SD that differs from the preset value by more than 10% may be explained by differences in the precision of gestational-age determination, the precision of the biometric measurement or biochemical assay or homogeneity of the screened population. If the observed SD is more than 10% smaller than the preset value, a change of setting would be appropriate. If the observed value is more than 10% larger, the potential causes need to be investigated before implementation of screening. If all four markers have a larger SD than the preset values, imprecision of gestational-age determination would be the most likely explanation and the results would be improved if a greater proportion were dated by ultrasound biometry. For the biophysical markers, differences in the results according to the person carrying out the Doppler examination, the person measuring blood pressure and the equipment might explain the findings. For the biochemical markers, internal and external quality indicators should be examined. Changes in the SD of a marker will have implications for the coefficients of correlation with other markers. The preset values for these parameters need to be multiplied by the ratio between the preset and observed SD. ASPRE adopted a risk cut-off of 1 in 100 for preterm PE but, in a locality with a different distribution of a-priori risk factors in the screened population, this may not be appropriate. This has been demonstrated using the combined data from KCH and SPREE with an example of screening with three markers (UtA-PI, MAP and PlGF)14. The overall detection and false-positive rates for preterm PE were 80% and 14%, respectively. However, the rates differed according to ethnicity; in Afro-Caribbean women, the rates were 92% and 33%, respectively, whilst in white women they were 69% and 10%. The risk cut-off most suitable for a given locality can be selected initially on the basis of a small series of screened women, such as those used to assess the normal median equations and SD. Once screening begins and more data accumulate, the selection can be re-evaluated and a new cut-off chosen if appropriate. A simple arithmetic method using the PE risks generated by the algorithm for each woman will yield reasonable estimates of the detection and false-positive rates for each risk cut-off. This method, although simple, is equivalent to statistical ‘modeling’ and, hence, does not require follow-up of the series to obtain the outcome of the pregnancy. The justification for the method is given in Appendix S1 and a worked example is shown in Table 1. All that is required is to download, or enter manually, the preterm PE risks from the series into a spreadsheet (Column A), with risk presented as 1 in n. These values are then sorted into descending order in Column B and expressed as a percentage in Column C. In Column D, each value in Column C is divided by the sum of Column C. A risk cut-off in Column B is chosen and all the values in Column D in cases with a risk higher than this cut-off are totaled; this is the estimated detection rate. In column E, 100% − C is calculated and, in Column F, this value is divided by the sum of Column E; the estimated false-positive rate is the total of the values in Column F in cases with a risk above the chosen cut-off. In the worked example in Table 1, using a 1 in 100 cut-off, the detection rate is 72% and the false-positive rate is 20%; raising the cut-off to 1 in 70, the rates are 59% and 12%, respectively. The methods of validation and calibration described here are similar to those that have been applied successfully to screening for Down syndrome. The use of statistical modelling to predict the performance of Down-syndrome screening protocols has proved to be a sound guide to prospective experience15. However, it could be argued that, unlike Down syndrome, PE is the result of multiple interactive causal factors that may not be accounted for fully in the risk-calculation algorithm. If so, performance could vary unpredictably between locations. Whilst this remains a theoretical possibility, there is no evidence for this in the literature. Four local single-center non-intervention studies of PE screening have been performed to assess directly detection and false-positive rates. They included 1129 pregnancies in Switzerland16, 617 in Brazil17, 3239 in Belgium18 and 541 in Norway19. The results were consistent with expectations from the large South London series, although they were too small to exclude a large deviation from expectation. For direct assessment, in addition to adequate size, a study protocol of high standards would be required. Apart from training in precise biophysical measurements and modification of preset software values, the protocol would need to include strict verification of PE cases and exclusion of women taking aspirin. This was a critical aspect of the FMF research program. It is unreliable to simply use hospital notification systems to identify cases and detailed examination of clinical and laboratory files is needed to distinguish PE from other forms of pregnancy-related hypertension. Intervention studies of PE screening are unable to assess directly the detection rate. Depending on aspirin dosage and compliance, this will necessarily be considerably lower than expected. Nonetheless, such studies can assess overall efficacy if they have a suitable control population. In Sydney, Australia, 2717 women were screened using UtA-PI, MAP and PAPP-A, and those at high risk were offered 150 mg aspirin/night20. The incidence of ‘early’ PE (requiring delivery before 34 weeks) was 0.04% compared with 0.4% in the 2-year period before screening began; for preterm PE, the rates were 0.37% and 0.83%, respectively. Both comparisons were statistically significant. In some medical centers, routine first-trimester UtA Doppler examination might not be possible. This is most likely to be the case in less well developed countries with limited resources. Given that the incidence of PE is particularly high in such localities, rather than not introducing screening at all, a simplified protocol could be considered. Using only the single marker maternal serum PlGF, the preterm PE detection rate based on the combined data from KCH and SPREE would be 61% at a false-positive rate of 10% and, with MAP as a second marker, this would increase to 66%14. For very early PE, the rates would be 72% and 79%, respectively, and, in low-resource countries, preventing such cases may be a key target for health planners. Data from ASPRE have been used to assess directly the impact of aspirin prophylaxis on length of stay in the neonatal intensive care unit21. Those in the aspirin arm of the trial who were admitted spent 11 days there compared with 31 days in the placebo arm, a statistically significant reduction, and 86% of the reduction in stay was due to very early deliveries. Advances in screening are usually slow to enter routine medical practice. Unlike classical medicine, in which patients present with symptoms and there is an imperative to act even when evidence of efficacy is weak, screening is proactive and hesitation is a reasonable response. However, today, the efficacy of PE screening is not in doubt and hesitation is no longer an option. Provided that local validation and calibration of risk calculation software are carried out, there is now no impediment to widespread implementation. H.C. is a consultant to PerkinElmer Inc, a provider of reagents for analytes used to predict PE, and a coholder of International Patent number PCT/US2009/032062 on ‘Methods for determining the risk of prenatal complications’. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.