Validity evidence for simulator-based obstetric ultrasound competency assessment tool: a multi-center study

Purpose: To collect validity evidence for a simulator-based obstetric ultrasound competency assessment tool (OUCAT). Methods: 89 sonographers from three centers (A, B, C), including novices (n=21), experienced trainees (n=44), and experts (n=24) participated in the competency assessment. Validity evidence of OUCAT was collected according to Standards for Educational and Psychological Testing. Content validity was ensured by reviewing guidelines and reaching expert consensus. Response process was ensured via training raters. Internal structure was explored through internal consistency, inter-rater reliability, and test-retest reliability. Relations to other variables were explored by comparing OUCAT scores of sonographers with different experience. Evidence for consequences was collected by determining the pass/fail level. Results: OUCAT included 123 items, of which 117 items could effectively distinguish novices from experts (P<0.05). The internal consistency was represented by the Cronbach’s α coefficient (0.978). The inter-rater reliability was high, with A being 0.868, B being 0.877, and C being 0.937 (P<0.001). Test-retest reliability was 0.732 (P=0.001). The performance of experts was significantly better than experienced trainees, and the performance of experienced trainees was significantly better than novices (70.3±10.7 vs 39.8±15.0 vs 20.5±10.6, P<0.001). The pass/fail level determined by contrast group method was 45 points. The passing rate of novices, experienced trainees and experts was 0% (0/21), 31.8% (14/44), and 100% (24/24) respectively. Conclusion: Simulator-based OUCAT exhibit good reliability and validity in assessing obstetric ultrasound skills.