title: Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors.
abstract: OBJECTIVE
  Despite great advances in medical image segmentation, the accurate and automatic segmentation of endoscopic scenes remains a challenging problem. Two important aspects have to be considered in segmenting an endoscopic scene: (1) noise and clutter due to light reflection and smoke from cutting tissue, and (2) structure occlusion (e.g. vessels occluded by fat, or endophytic tumours occluded by healthy kidney tissue).
  
  
  METHODS
  In this paper, we propose a variational technique to augment a surgeon's endoscopic view by segmenting visible as well as occluded structures in the intraoperative endoscopic view. Our method estimates the 3D pose and deformation of anatomical structures segmented from 3D preoperative data in order to align to and segment corresponding structures in 2D intraoperative endoscopic views. Our preoperative to intraoperative alignment is driven by, first, spatio-temporal, signal processing based vessel pulsation cues and, second, machine learning based analysis of colour and textural visual cues. To our knowledge, this is the first work that utilizes vascular pulsation cues for guiding preoperative to intraoperative registration. In addition, we incorporate a tissue-specific (i.e. heterogeneous) physically based deformation model into our framework to cope with the non-rigid deformation of structures that occurs during the intervention.
  
  
  RESULTS
  We validated the utility of our technique on fifteen challenging clinical cases with 45 % improvements in accuracy compared to the state-of-the-art method.
  
  
  CONCLUSIONS
  A new technique for localizing both visible and occluded structures in an endoscopic view was proposed and tested. This method leverages both preoperative data, as a source of patient-specific prior knowledge, as well as vasculature pulsation and endoscopic visual cues in order to accurately segment the highly noisy and cluttered environment of an endoscopic video. Our results on in vivo clinical cases of partial nephrectomy illustrate the potential of the proposed framework for augmented reality applications in minimally invasive surgeries.
authors.affiliation: Medical Image Analysis Lab, Simon Fraser University, Burnaby, BC, V5A 1S6, Canada. smasoudn@gmail.com.
authors.forename: Masoud S
authors.initials: MS
authors.lastname: Nosrati
authors.affiliation: BiSICL, University of British Columbia, Vancouver, BC, Canada.
authors.forename: Alborz
authors.initials: A
authors.lastname: Amir-Khalili
authors.affiliation: Qatar Robotic Surgery Centre, Qatar Science and Technology Park, Doha, Qatar.
authors.forename: Jean-Marc
authors.initials: JM
authors.lastname: Peyrat
authors.affiliation: Qatar Robotic Surgery Centre, Qatar Science and Technology Park, Doha, Qatar.
authors.forename: Julien
authors.initials: J
authors.lastname: Abinahed
authors.affiliation: Urology Department, Hamad General Hospital, Hamad Medical Corporation, Doha, Qatar.
authors.forename: Osama
authors.initials: O
authors.lastname: Al-Alao
authors.affiliation: Urology Department, Hamad General Hospital, Hamad Medical Corporation, Doha, Qatar.
authors.forename: Abdulla
authors.initials: A
authors.lastname: Al-Ansari
authors.affiliation: BiSICL, University of British Columbia, Vancouver, BC, Canada.
authors.forename: Rafeef
authors.initials: R
authors.lastname: Abugharbieh
authors.affiliation: Medical Image Analysis Lab, Simon Fraser University, Burnaby, BC, V5A 1S6, Canada.
authors.forename: Ghassan
authors.initials: G
authors.lastname: Hamarneh
mesh_terms: D003116:Color; D004724:Endoscopy / Q000379:methods*; D006801:Humans; D021621:Imaging, Three-Dimensional / Q000379:methods*; D009392:Nephrectomy / Q000379:methods
keywords: 3D pose estimation; Endoscopy; Image-guided surgery; Kidney; Occluded vessels; Partial nephrectomy; Patient-specific model; Robotic surgery; Segmentation