title: Assessment of resident operative performance using a real-time mobile Web system: preparing for the milestone age.
abstract: OBJECTIVE
  To satisfy trainees' operative competency requirements while improving feedback validity and timeliness using a mobile Web-based platform.
  
  
  METHODS
  The Southern Illinois University Operative Performance Rating Scale (OPRS) was embedded into a website formatted for mobile devices. From March 2013 to February 2014, faculty members were instructed to complete the OPRS form while providing verbal feedback to the operating resident at the conclusion of each procedure. Submitted data were compiled automatically within a secure Web-based spreadsheet. Conventional end-of-rotation performance (CERP) evaluations filed 2006 to 2013 and OPRS performance scores were compared by year of training using serial and independent-samples t tests. The mean CERP scores and OPRS overall resident operative performance scores were directly compared using a linear regression model. OPRS mobile site analytics were reviewed using a Web-based reporting program.
  
  
  METHODS
  Large university-based general surgery residency program.
  
  
  METHODS
  General Surgery faculty used the mobile Web OPRS system to rate resident performance. Residents and the program director reviewed evaluations semiannually.
  
  
  RESULTS
  Over the study period, 18 faculty members and 37 residents logged 176 operations using the mobile OPRS system. There were 334 total OPRS website visits. Median time to complete an evaluation was 45 minutes from the end of the operation, and faculty spent an average of 134 seconds on the site to enter 1 assessment. In the 38,506 CERP evaluations reviewed, mean performance scores showed a positive linear trend of 2% change per year of training (p = 0.001). OPRS overall resident operative performance scores showed a significant linear (p = 0.001), quadratic (p = 0.001), and cubic (p = 0.003) trend of change per year of clinical training, reflecting the resident operative experience in our training program. Differences between postgraduate year-1 and postgraduate year-5 overall performance scores were greater with the OPRS (mean = 0.96, CI: 0.55-1.38) than with CERP measures (mean = 0.37, CI: 0.34-0.41). Additionally, there were consistent increases in each of the OPRS subcategories.
  
  
  CONCLUSIONS
  In contrast to CERPs, the OPRS fully satisfies the Accreditation Council for Graduate Medical Education and American Board of Surgery operative assessment requirements. The mobile Web platform provides a convenient interface, broad accessibility, automatic data compilation, and compatibility with common database and statistical software. Our mobile OPRS system encourages candid feedback dialog and generates a comprehensive review of individual and group-wide operative proficiency in real time.
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California. Electronic address: jwagner@mednet.ucla.edu.
authors.forename: Justin P
authors.initials: JP
authors.lastname: Wagner
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California.
authors.forename: David C
authors.initials: DC
authors.lastname: Chen
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California.
authors.forename: Timothy R
authors.initials: TR
authors.lastname: Donahue
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California.
authors.forename: Chi
authors.initials: C
authors.lastname: Quach
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California.
authors.forename: O Joe
authors.initials: OJ
authors.lastname: Hines
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California.
authors.forename: Jonathan R
authors.initials: JR
authors.lastname: Hiatt
authors.affiliation: David Geffen School of Medicine, University of California, Los Angeles, California.
authors.forename: Areti
authors.initials: A
authors.lastname: Tillou
mesh_terms: D000328:Adult; D002983:Clinical Competence; D005260:Female; D013502:General Surgery / Q000193:education*; D006801:Humans; D020407:Internet; D007396:Internship and Residency; D007721:Knowledge of Results, Psychological; D016014:Linear Models; D008297:Male
keywords: Interpersonal and Communication Skills; Milestones; Patient Care; Practice-Based Learning and Improvement; Professionalism; evaluations; feedback; operative performance; residents; surgical skills