Development and validation of a new tool for assessment of trainees’ interventional musculoskeletal ultrasound skills

Abstract Objectives Interventional musculoskeletal ultrasound (MSUS) procedures are routinely performed in rheumatology practice. However, the efficacy and safety of the procedures rely on the competence of the physician, and assessment of skills is crucial. Thus, this study aimed to develop and establish validity evidence for a tool assessing trainees' interventional MSUS skills. Methods An expert panel of rheumatologists modified an existing tool for assessing competences in invasive abdominal and thoracic ultrasound procedures. The new tool (the Assessment of Interventional Musculoskeletal Ultrasound Skills (AIMUS) tool) reflects the essential steps in interventional MSUS. To establish validity evidence, physicians with different levels of interventional MSUS experience were enrolled and performed two procedures on a rubber fantom, simulating real patient cases. All performances were video recorded, anonymized, and assessed in random order by two blinded raters using the AIMUS tool. Results Sixty-five physicians from 21 different countries were included and categorized into groups based on their experience, resulting in 130 videos for analysis. The internal consistency of the tool was excellent, with a Cronbach’s alpha of 0.96. The inter-case reliability was good with a Pearson’s correlation coefficient (PCC) of 0.74 and the inter-rater reliability was moderate to good (PCC 0.58). The ability to discriminate between different levels of experience was highly significant (p< 0.001). Conclusions We have developed and established validity evidence for a new interventional MSUS assessment tool. The tool can be applied in future competency-based educational programs, provide structured feedback to trainees in daily clinical practice, and ensure end-of-training competence.