Methodologies for establishing validity in surgical simulation studies.
BACKGROUND
Validating assessment tools in surgical simulation training is critical to objectively measuring skills. Most reviews do not elicit methodologies for conducting rigorous validation studies. Our study reports current methodological approaches and proposes benchmark criteria for establishing validity in surgical simulation studies.


METHODS
We conducted a systematic review of studies establishing validity. A PubMed search was performed with the following keywords: "validity/validation," "simulation," "surgery," and "technical skills." Descriptors were tabulated for 29 methodological variables by 2 reviewers.


RESULTS
A total of 83 studies were included in the review. Of these studies, 60% targeted construct, 24% targeted concurrent, and 5% looked at predictive validity. Less than half (45%) of all the studies reported reliability data. Most studies (82%) were conducted in a single institution with a mean of 37 subjects recruited. Only half of the studies provided rationale for task selection. Data sources included simulator-generated measures (34%), performance assessment by human evaluators (33%), motion tracking (6%), and combined modes (28%). In studies using human evaluators, videotaping was a common (48%) blinding technique; however, 34% of the studies did not blind evaluators. Commonly reported outcomes included task time (86%), economy of motion (51%), technical errors (48%), and number of movements (25%).


CONCLUSIONS
The typical validation study comes from a single institution with a small sample size, lacks clear justification for task selection, omits reliability reporting, and poses potential bias in study design. The lack of standardized validation methodologies creates challenges for training centers that survey the literature to determine the appropriate method for their local settings.
Van Nortwick
Sara S
SS
Institute for Simulation and Interprofessional Studies (ISIS), Seattle Children's, University of Washington School of Medicine, Seattle, WA, USA.
Lendvay
Thomas S
TS
Jensen
Aaron R
AR
Wright
Andrew S
AS
Horvath
Karen D
KD
Kim
Sara
S
D003162:Competency-Based Education / Q000379:methods / Q000592:standards; D003194:Computer-Assisted Instruction / Q000379:methods* / Q000592:standards*; D004503:Education, Medical, Graduate / Q000379:methods* / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D013502:General Surgery / Q000193:education*; D006801:Humans; D015203:Reproducibility of Results