title: Histology image search using multimodal fusion.
abstract: This work proposes a histology image indexing strategy based on multimodal representations obtained from the combination of visual features and associated semantic annotations. Both data modalities are complementary information sources for an image retrieval system, since visual features lack explicit semantic information and semantic terms do not usually describe the visual appearance of images. The paper proposes a novel strategy to build a fused image representation using matrix factorization algorithms and data reconstruction principles to generate a set of multimodal features. The methodology can seamlessly recover the multimodal representation of images without semantic annotations, allowing us to index new images using visual features only, and also accepting single example images as queries. Experimental evaluations on three different histology image data sets show that our strategy is a simple, yet effective approach to building multimodal representations for histology image search, and outperforms the response of the popular late fusion approach to combine information.
authors.affiliation: University of Illinois at Urbana-Champaign, Sieble Center for Computer Science, 201 N Goodwin Ave, Urbana, IL 61801, USA. Electronic address: caicedo@illinois.edu.
authors.forename: Juan C
authors.initials: JC
authors.lastname: Caicedo
authors.affiliation: MindLab Research Laboratory, Universidad Nacional de Colombia, Bogotá, Colombia. Electronic address: javanegasr@unal.edu.co.
authors.forename: Jorge A
authors.initials: JA
authors.lastname: Vanegas
authors.affiliation: MindLab Research Laboratory, Universidad Nacional de Colombia, Bogotá, Colombia. Electronic address: fmpaezri@unal.edu.co.
authors.forename: Fabian
authors.initials: F
authors.lastname: Páez
authors.affiliation: MindLab Research Laboratory, Universidad Nacional de Colombia, Bogotá, Colombia. Electronic address: fagonzalezo@unal.edu.co.
authors.forename: Fabio A
authors.initials: FA
authors.lastname: González
mesh_terms: D000465:Algorithms; D001706:Biopsy / Q000379:methods; D057225:Data Mining / Q000379:methods*; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008853:Microscopy / Q000379:methods*; D064847:Multimodal Imaging / Q000379:methods*; D009323:Natural Language Processing; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D011873:Radiology Information Systems / Q000458:organization & administration*; D015203:Reproducibility of Results; D012660:Semantics; D012680:Sensitivity and Specificity; D013382:Subtraction Technique
keywords: Digital pathology; Histology; Image search; Multimodal fusion; Semantic spaces; Visual representation