title: Establishing Benchmarks For Minimum Competence With Dry Lab Robotic Surgery Drills.
abstract: The Robotic Objective Structured Assessment of Technical Skills (R-OSATS) is a previously validated assessment tool that is used to assess 5 standardized inanimate robotic surgery drills. R-OSATS is used to evaluate performance on surgical drills, with scores of 0 to 20 for each drill. Our objective was to establish the minimum threshold score that denotes competence on these drills. Thus, we performed a standard setting study using data from surgeons and trainees in 8 academic medical centers. Cutoff scores for the minimal level of competence using R-OSATS were established using 2 techniques: the modified Angoff and the contrasting groups methods. For the modified Angoff method, 8 content experts met and, in an iterative process, derived the scores that a minimally competent trainee should receive. After 2 iterative rounds of scoring and discussion with the modified Angoff method, we established a minimum competence score per drill with high agreement (rWG range, 0.92-0.98). There was unanimous consensus that a trainee needs to achieve competence on each independent drill. A second method, the contrasting groups method, was used to verify our results. In this method, we compared R-OSATS scores from "inexperienced" (34 postgraduate year 1 and 2 trainees) with "experienced" (22 faculty and fellow) robotic surgeons. The distributions of scores from both groups were plotted, and a cutoff score for each drill was determined from the intersection of the 2 curves. Using this method, the minimum score for competence would be 14 per drill, which is slightly more stringent but confirms the results obtained from the modified Angoff approach. In conclusion, using 2 well-described standard setting techniques, we have established minimum benchmarks designating trainee competence for 5 dry lab robotic surgery drills.
authors.affiliation: Department of Obstetrics and Gynecology, Duke University, Durham, North Carolina. Electronic address: nazema.siddiqui@duke.edu.
authors.forename: Nazema Y
authors.initials: NY
authors.lastname: Siddiqui
authors.affiliation: Department of Obstetrics and Gynecology, University of North Carolina-Charlotte Campus, Charlotte, North Carolina.
authors.forename: Megan E
authors.initials: ME
authors.lastname: Tarr
authors.affiliation: Department of Obstetrics and Gynecology, University of North Carolina, Chapel Hill, North Carolina.
authors.forename: Elizabeth J
authors.initials: EJ
authors.lastname: Geller
authors.affiliation: Department of Obstetrics and Gynecology, Columbia University Medical Center, New York, New York.
authors.forename: Arnold P
authors.initials: AP
authors.lastname: Advincula
authors.affiliation: Department of Obstetrics and Gynecology, Wright State University, Dayton, Ohio.
authors.forename: Michael L
authors.initials: ML
authors.lastname: Galloway
authors.affiliation: Department of Obstetrics and Gynecology, Johns Hopkins University, Baltimore, Maryland.
authors.forename: Isabel C
authors.initials: IC
authors.lastname: Green
authors.affiliation: Department of Obstetrics and Gynecology, Beth Israel Deaconess Medical Center, Boston, Massachusetts.
authors.forename: Hye-Chun
authors.initials: HC
authors.lastname: Hur
authors.affiliation: Department of Obstetrics and Gynecology, Columbia University Medical Center, New York, New York.
authors.forename: Michael C
authors.initials: MC
authors.lastname: Pitter
authors.affiliation: Department of Biostatistics and Bioinformatics, Duke University, Durham, North Carolina.
authors.forename: Emily E
authors.initials: EE
authors.lastname: Burke
authors.affiliation: Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania.
authors.forename: Martin A
authors.initials: MA
authors.lastname: Martino
mesh_terms: D019985:Benchmarking / Q000592:standards*; D002983:Clinical Competence / Q000592:standards*; D019317:Evidence-Based Medicine; D005240:Feasibility Studies; D006801:Humans; D065287:Robotic Surgical Procedures / Q000592:standards*; D012371:Robotics / Q000592:standards; D066231:Surgeons / Q000592:standards*
keywords: Objective Structured Assessment of Technical Skills; Robotic Objective Structured Assessment of Technical Skills; Robotic surgery; Simulation; Surgical education; Technical skills