Assessing adverse event reports of hysteroscopic sterilization device removal using natural language processing

Abstract Objective To develop an annotation model to apply natural language processing (NLP) to device adverse event reports and implement the model to evaluate the most frequently experienced events among women reporting a sterilization device removal. Methods We included adverse event reports from the Manufacturer and User Facility Device Experience database from January 2005 to June 2018 related to device removal following hysteroscopic sterilization. We used an iterative process to develop an annotation model that extracts six categories of desired information and applied the annotation model to train an NLP algorithm. We assessed the model performance using positive predictive value (PPV, also known as precision), sensitivity (also known as recall), and F1 score (a combined measure of PPV and sensitivity). Using extracted variables, we summarized the reporting source, the presence of prespecified and other patient and device events, additional sterilizations and other procedures performed, and time from implantation to removal. Results The overall F1 score was 91.5% for labeled items and 93.9% for distinct events after excluding duplicates. A total of 16 535 reports of device removal were analyzed. The most frequently reported patient and device events were abdominal/pelvic/genital pain (N = 13 166, 79.6%) and device dislocation/migration (N = 3180, 19.2%), respectively. Of those reporting an additional sterilization procedure, the majority had a hysterectomy or salpingectomy (N = 7932). One‐fifth of the cases that had device removal timing specified reported a removal after 7 years following implantation (N = 2444/11 293). Conclusions We present a roadmap to develop an annotation model for NLP to analyze device adverse event reports. The extracted information is informative and complements findings from previous research using administrative data.