title: Reliability of laparoscopic skills assessment on video: 8-year results of the endoscopic surgical skill qualification system in Japan.
abstract: OBJECTIVE
  The Japanese Urological Association and Japanese Society of Endourology established a urologic laparoscopic skills qualification system called the Endoscopic Surgical Skill Qualification (ESSQ) System in Urological Laparoscopy in 2004. The reliability of video assessments by referees was evaluated.
  
  
  METHODS
  Videos of nephrectomies or adrenalectomies performed by the applicants were assessed by two referees selected among a pool of 42 referees. From 2004 to 2011, 1308 urologists applied and 60.2% were qualified after video assessments. The results of skills assessments on 1220 videos that had fixed points by two referees were analyzed statistically.
  
  
  RESULTS
  The average number of videos that each referee assessed was 58.1, with a range of 16 to 87. The accordance rate of the results of the video assessment, pass or fail, by the two referees was 68.9%. The scores of the video assessment by each referee averaged 62.7±2.4 (standard deviation) (full score was set at 75 points and ≥60 points was needed to pass). There was a statistically significant difference in the average video assessment score among the referees (P<0.001), and five referees showed significantly higher or lower average scores than the other referees. The percentage qualification of the final decision made by the Referee Committee on the videos originally assessed by each referee showed no significant differences among the 42 referees. The accordance rate of the results from the video assessment by each referee with the final decision by the committee showed a statistically significant positive correlation with the number of videos assessed by each referee (r=0.404, P=0.0080).
  
  
  CONCLUSIONS
  The ESSQ system showed moderate reliability for the video assessments by the referees. It was concluded that the video assessments by the referees were fair for all applicants, because the final qualification rates showed no significant differences among the referees.
authors.affiliation: 1 Department of Urology and Andrology, Kansai Medical University , Hirakata, Japan .
authors.forename: Tadashi
authors.initials: T
authors.lastname: Matsuda
authors.forename: Hiroomi
authors.initials: H
authors.lastname: Kanayama
authors.forename: Yoshinari
authors.initials: Y
authors.lastname: Ono
authors.forename: Akihiro
authors.initials: A
authors.lastname: Kawauchi
authors.forename: Hiroaki
authors.initials: H
authors.lastname: Mizoguchi
authors.forename: Ken
authors.initials: K
authors.lastname: Nakagawa
authors.forename: Masatsugu
authors.initials: M
authors.lastname: Iwamura
authors.forename: Masanobu
authors.initials: M
authors.lastname: Shigeta
authors.forename: Tomonori
authors.initials: T
authors.lastname: Habuchi
authors.forename: Toshiro
authors.initials: T
authors.lastname: Terachi
mesh_terms: D000315:Adrenalectomy / Q000592:standards*; D000704:Analysis of Variance; D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D004724:Endoscopy; D006801:Humans; D007564:Japan; D010535:Laparoscopy / Q000193:education / Q000592:standards*; D009392:Nephrectomy / Q000592:standards*; D015588:Observer Variation; D015203:Reproducibility of Results; D014572:Urology; D014741:Video Recording