title: Objective assessment of operator performance during ultrasound-guided procedures.
abstract: OBJECTIVE
  Simulation permits objective assessment of operator performance in a controlled and safe environment. Image-guided procedures often require accurate needle placement, and we designed a system to monitor how ultrasound guidance is used to monitor needle advancement toward a target. The results were correlated with other estimates of operator skill.
  
  
  METHODS
  The simulator consisted of a tissue phantom, ultrasound unit, and electromagnetic tracking system. Operators were asked to guide a needle toward a visible point target. Performance was video-recorded and synchronized with the electromagnetic tracking data. A series of algorithms based on motor control theory and human information processing were used to convert raw tracking data into different performance indices. Scoring algorithms converted the tracking data into efficiency, quality, task difficulty, and targeting scores that were aggregated to create performance indices. After initial feasibility testing, a standardized assessment was developed. Operators (N = 12) with a broad spectrum of skill and experience were enrolled and tested. Overall scores were based on performance during ten simulated procedures. Prior clinical experience was used to independently estimate operator skill.
  
  
  RESULTS
  When summed, the performance indices correlated well with estimated skill. Operators with minimal or no prior experience scored markedly lower than experienced operators. The overall score tended to increase according to operator's clinical experience. Operator experience was linked to decreased variation in multiple aspects of performance. The aggregated results of multiple trials provided the best correlation between estimated skill and performance. A metric for the operator's ability to maintain the needle aimed at the target discriminated between operators with different levels of experience.
  
  
  CONCLUSIONS
  This study used a highly focused task model, standardized assessment, and objective data analysis to assess performance during simulated ultrasound-guided needle placement. The performance indices were closely related to operator experience.
authors.affiliation: Rush Medical College, Chicago, IL, USA.
authors.forename: David M
authors.initials: DM
authors.lastname: Tabriz
authors.forename: Mandie
authors.initials: M
authors.lastname: Street
authors.forename: Thomas K
authors.initials: TK
authors.lastname: Pilgram
authors.forename: James R
authors.initials: JR
authors.lastname: Duncan
mesh_terms: D003198:Computer Simulation; D055590:Electromagnetic Phenomena; D005069:Evaluation Studies as Topic; D006801:Humans; D017145:Models, Educational; D016343:Monitoring, Intraoperative / Q000379:methods; D015397:Program Evaluation / Q000379:methods*; D012680:Sensitivity and Specificity; D025321:Surgery, Computer-Assisted / Q000379:methods*; D013647:Task Performance and Analysis; D018084:Ultrasonography, Interventional / Q000592:standards*; D014741:Video Recording