title: Multi-sensor super-resolution for hybrid range imaging with application to 3-D endoscopy and open surgery.
abstract: In this paper, we propose a multi-sensor super-resolution framework for hybrid imaging to super-resolve data from one modality by taking advantage of additional guidance images of a complementary modality. This concept is applied to hybrid 3-D range imaging in image-guided surgery, where high-quality photometric data is exploited to enhance range images of low spatial resolution. We formulate super-resolution based on the maximum a-posteriori (MAP) principle and reconstruct high-resolution range data from multiple low-resolution frames and complementary photometric information. Robust motion estimation as required for super-resolution is performed on photometric data to derive displacement fields of subpixel accuracy for the associated range images. For improved reconstruction of depth discontinuities, a novel adaptive regularizer exploiting correlations between both modalities is embedded to MAP estimation. We evaluated our method on synthetic data as well as ex-vivo images in open surgery and endoscopy. The proposed multi-sensor framework improves the peak signal-to-noise ratio by 2 dB and structural similarity by 0.03 on average compared to conventional single-sensor approaches. In ex-vivo experiments on porcine organs, our method achieves substantial improvements in terms of depth discontinuity reconstruction.
authors.affiliation: Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany; Erlangen Graduate School in Advanced Optical Technologies (SAOT), Germany. Electronic address: thomas.koehler@fau.de.
authors.forename: Thomas
authors.initials: T
authors.lastname: Köhler
authors.affiliation: Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
authors.forename: Sven
authors.initials: S
authors.lastname: Haase
authors.affiliation: Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
authors.forename: Sebastian
authors.initials: S
authors.lastname: Bauer
authors.affiliation: Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
authors.forename: Jakob
authors.initials: J
authors.lastname: Wasza
authors.affiliation: Division of Medical and Biological Informatics Junior Group: Computer-assisted Interventions, German Cancer Research Center (DKFZ) Heidelberg, Germany.
authors.forename: Thomas
authors.initials: T
authors.lastname: Kilgus
authors.affiliation: Division of Medical and Biological Informatics Junior Group: Computer-assisted Interventions, German Cancer Research Center (DKFZ) Heidelberg, Germany.
authors.forename: Lena
authors.initials: L
authors.lastname: Maier-Hein
authors.affiliation: Institute of Medical Biometry and Informatics, University of Heidelberg, Germany.
authors.forename: Christian
authors.initials: C
authors.lastname: Stock
authors.affiliation: Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany; Erlangen Graduate School in Advanced Optical Technologies (SAOT), Germany.
authors.forename: Joachim
authors.initials: J
authors.lastname: Hornegger
authors.affiliation: Research Group Minimally-invasive interdisciplinary therapeutical intervention, Klinikum rechts der Isar of the Technical University Munich, Germany.
authors.forename: Hubertus
authors.initials: H
authors.lastname: Feußner
mesh_terms: D000465:Algorithms; D004724:Endoscopy / Q000379:methods*; D006801:Humans; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods; D021621:Imaging, Three-Dimensional / Q000379:methods*; D064847:Multimodal Imaging / Q000379:methods*; D010783:Photometry / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013382:Subtraction Technique; D025321:Surgery, Computer-Assisted / Q000379:methods*
keywords: 3-D endoscopy; Hybrid range imaging; Open surgery; Sensor fusion; Super-resolution