title: Query-by-example surgical activity detection.
abstract: OBJECTIVE
  Easy acquisition of surgical data opens many opportunities to automate skill evaluation and teaching. Current technology to search tool motion data for surgical activity segments of interest is limited by the need for manual pre-processing, which can be prohibitive at scale. We developed a content-based information retrieval method, query-by-example (QBE), to automatically detect activity segments within surgical data recordings of long duration that match a query.
  
  
  METHODS
  The example segment of interest (query) and the surgical data recording (target trial) are time series of kinematics. Our approach includes an unsupervised feature learning module using a stacked denoising autoencoder (SDAE), two scoring modules based on asymmetric subsequence dynamic time warping (AS-DTW) and template matching, respectively, and a detection module. A distance matrix of the query against the trial is computed using the SDAE features, followed by AS-DTW combined with template scoring, to generate a ranked list of candidate subsequences (substrings). To evaluate the quality of the ranked list against the ground-truth, thresholding conventional DTW distances and bipartite matching are applied. We computed the recall, precision, F1-score, and a Jaccard index-based score on three experimental setups. We evaluated our QBE method using a suture throw maneuver as the query, on two tool motion datasets (JIGSAWS and MISTIC-SL) captured in a training laboratory.
  
  
  RESULTS
  We observed a recall of 93, 90 and 87 % and a precision of 93, 91, and 88 % with same surgeon same trial (SSST), same surgeon different trial (SSDT) and different surgeon (DS) experiment setups on JIGSAWS, and a recall of 87, 81 and 75 % and a precision of 72, 61, and 53 % with SSST, SSDT and DS experiment setups on MISTIC-SL, respectively.
  
  
  CONCLUSIONS
  We developed a novel, content-based information retrieval method to automatically detect multiple instances of an activity within long surgical recordings. Our method demonstrated adequate recall across different complexity datasets and experimental conditions.
authors.affiliation: Department of Computer Science, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA. yxgao@jhu.edu.
authors.forename: Yixin
authors.identifier: http://orcid.org/0000-0001-8854-8959
authors.initials: Y
authors.lastname: Gao
authors.affiliation: Department of Computer Science, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA.
authors.forename: S Swaroop
authors.initials: SS
authors.lastname: Vedula
authors.affiliation: Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, MD, 21287, USA.
authors.forename: Gyusung I
authors.initials: GI
authors.lastname: Lee
authors.affiliation: Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, MD, 21287, USA.
authors.forename: Mija R
authors.initials: MR
authors.lastname: Lee
authors.affiliation: Department of Electrical and Computer Engineering, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA.
authors.forename: Sanjeev
authors.initials: S
authors.lastname: Khudanpur
authors.affiliation: Department of Computer Science, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD, 21218, USA.
authors.forename: Gregory D
authors.initials: GD
authors.lastname: Hager
mesh_terms: D000465:Algorithms; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*; D013514:Surgical Procedures, Operative
keywords: Asymmetric subsequence dynamic time warping; Query-by-example; Stacked denoising autoencoder; Surgical activity detection; Surgical data indexing