WE3.9 Operative competence should be assessed on a procedure-specific basis: analysis of submitted Supervised Structured Assessments of Operative Performance (SSAOPs) by Core Surgical Trainees in Ireland using Generalisability Theory

Abstract Introduction Competency-based training programmes require reliable summative assessment methods. The aim of this study is to assess the reliability of the Supervised Structured Assessment of Operative Performance (SSAOP) tool in determining the operative competence of core surgical trainees in Ireland. Methods SSAOP scores from April 2016 - February 2021 were analysed. Reliability analyses for assessment of laparoscopic appendicectomy were compared to those across all submitted procedural assessments. Generalizability and decision studies determined the number of assessments and observers needed to achieve a reliability coefficient (G) of ≥0.7 and ≥0.8 (appropriate for low- and high-stakes assessment respectively). Results 2,294 SSAOPs were analysed; 85 were completed for laparoscopic appendicectomy. G ≥0.70 is achieved using a single assessor observing a minimum of 3 laparoscopic appendicectomy cases, though G ≥0.70 cannot be achieved for assessments across a wider group of procedures without large numbers of assessors and cases. G ≥0.80 can be achieved with a minimum of two random assessors each observing 3 cases using the Overall Performance rating, or by two assessors and 6 assessments using the ‘Total Checklist’ score. Reliable assessment of a trainee's skill across all procedures, regardless of sub-specialty, could only be achieved using 89 cases and 7 assessors using the Overall Performance rater, or 67 cases and 9 assessors using the Total Checklist. Conclusion Operative competence should be assessed on a procedure-specific basis. Assessing overall competence across procedures is not feasible in practice. Trainers and trainees should focus on repeated assessments observing index procedures suitable for a given trainee's stage of training.