title: Content-based image retrieval in radiology: analysis of variability in human perception of similarity.
abstract: We aim to develop a better understanding of perception of similarity in focal computed tomography (CT) liver images to determine the feasibility of techniques for developing reference sets for training and validating content-based image retrieval systems. In an observer study, four radiologists and six nonradiologists assessed overall similarity and similarity in 5 image features in 136 pairs of focal CT liver lesions. We computed intra- and inter-reader agreements in these similarity ratings and viewed the distributions of the ratings. The readers' ratings of overall similarity and similarity in each feature primarily appeared to be bimodally distributed. Median Kappa scores for intra-reader agreement ranged from 0.57 to 0.86 in the five features and from 0.72 to 0.82 for overall similarity. Median Kappa scores for inter-reader agreement ranged from 0.24 to 0.58 in the five features and were 0.39 for overall similarity. There was no significant difference in agreement for radiologists and nonradiologists. Our results show that developing perceptual similarity reference standards is a complex task. Moderate to high inter-reader variability precludes ease of dividing up the workload of rating perceptual similarity among many readers, while low intra-reader variability may make it possible to acquire large volumes of data by asking readers to view image pairs over many sessions.
authors.affiliation: Stanford University , Department of Electrical Engineering, 350 Serra Mall, Stanford, California 94305, United States.
authors.forename: Jessica
authors.initials: J
authors.lastname: Faruque
authors.affiliation: Stanford University Medical Center , Department of Radiology, 300 Pasteur Drive, Room S078, MC 5105, Stanford, California 94305, United States.
authors.forename: Christopher F
authors.initials: CF
authors.lastname: Beaulieu
authors.affiliation: Stanford University , Department of Radiology, Lucas MRS Imaging Center, 1201 Welch Road, Room P-280, Stanford, California 94305-5488, United States.
authors.forename: Jarrett
authors.initials: J
authors.lastname: Rosenberg
authors.affiliation: Stanford University , Departments of Radiology and Medicine (Biomedical Informatics), Richard M. Lucas Center P285, 1201 Welch Road, Stanford, California 94305-5488, United States.
authors.forename: Daniel L
authors.initials: DL
authors.lastname: Rubin
authors.affiliation: Stanford University , Department of Radiology, 3801 Miranda Avenue, Palo Alto, California 94304-1290, United States.
authors.forename: Dorcas
authors.initials: D
authors.lastname: Yao
authors.affiliation: Stanford University , Department of Radiology, James H. Clark Center, 318 Campus Drive, W3.1, Stanford, California 94305-5441, United States.
authors.forename: Sandy
authors.initials: S
authors.lastname: Napel
keywords: computed tomography; content-based image retrieval; liver; perception; reference standards; similarity