title: Objective assessment of robotic surgical skill using instrument contact vibrations.
abstract: BACKGROUND
  Surgical skill evaluation ordinarily requires tedious video review and survey completion, while new automatic approaches focus on evaluating the quality of the surgeon's movements in free space. Robotic surgical instrument vibrations are simple to measure and physically correspond to how roughly instruments are handled, but they have yet to be studied as a measure of technical surgical skill.
  
  
  METHODS
  Thirteen surgeons used a robotic surgery system (da Vinci S by Intuitive Surgical) to perform four trials each of peg transfer (PT), needle pass (NP), and intracorporeal suturing (IS). Completion time, instrument vibrations, and applied forces were measured for each trial; root mean square (RMS) and total sum of squares (TSS) were calculated from both the vibration and force recordings. Four experienced surgeons blindly assessed the task videos using a Global Rating Scale (GRS), and skill metrics were compared between the eight novices and five experienced participants. Stepwise regression was performed to predict GRS score from objective skill metrics. The concurrent validity of each metric was evaluated using receiver operating characteristic (ROC) analysis.
  
  
  RESULTS
  The GRS demonstrated excellent internal consistency (Cronbach's α = 0.91) and strong inter-rater reliability (ICC = 0.84). Compared to novices, experienced surgeons earned higher GRS scores and performed tasks with lower vibration magnitudes, lower forces, and shorter completion times in 15 of 18 task-metric combinations (p values ranging from 0.042 to <0.001). ROC analysis demonstrated that including vibration and force magnitudes along with completion time in skill prediction models improves the objective classification of subjects as novice or experienced for all tasks studied (PT: 90% sensitivity, 75% specificity; NP: 85% sensitivity, 84% specificity; suturing: 100% sensitivity, 100% specificity).
  
  
  CONCLUSIONS
  RMS and TSS instrument vibrations are novel construct-valid measures of robotic surgical skill that enable the development of objective skill assessment models comparable to observer-based ratings.
authors.affiliation: Department of Otorhinolaryngology - Head and Neck Surgery, Hospital of the University of Pennsylvania, Philadelphia, PA, USA. ernest.gomez@uphs.upenn.edu.|Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, 224 Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, USA. ernest.gomez@uphs.upenn.edu.
authors.forename: Ernest D
authors.initials: ED
authors.lastname: Gomez
authors.affiliation: Department of Surgery, McGill University Health Centre, Philadelphia, PA, USA.|Arnold and Blema Steinberg Medical Simulation Centre, Philadelphia, PA, USA.
authors.forename: Rajesh
authors.initials: R
authors.lastname: Aggarwal
authors.affiliation: Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, 224 Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, USA.
authors.forename: William
authors.initials: W
authors.lastname: McMahan
authors.affiliation: Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, 224 Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, USA.
authors.forename: Karlin
authors.initials: K
authors.lastname: Bark
authors.affiliation: Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, 224 Towne Building, 220 South 33rd Street, Philadelphia, PA, 19104, USA. kuchenbe@seas.upenn.edu.
authors.forename: Katherine J
authors.initials: KJ
authors.lastname: Kuchenbecker
mesh_terms: D000328:Adult; D002983:Clinical Competence; D004867:Equipment Design; D005260:Female; D006801:Humans; D012372:ROC Curve; D015203:Reproducibility of Results; D012371:Robotics / Q000295:instrumentation*; D066231:Surgeons / Q000592:standards*; D014732:Vibration
keywords: Assessment; Objective; Robotic; Skill; Vibrations