title: Modeling perceptual similarity measures in CT images of focal liver lesions.
abstract: BACKGROUND
  A gold standard for perceptual similarity in medical images is vital to content-based image retrieval, but inter-reader variability complicates development. Our objective was to develop a statistical model that predicts the number of readers (N) necessary to achieve acceptable levels of variability.
  
  
  METHODS
  We collected 3 radiologists' ratings of the perceptual similarity of 171 pairs of CT images of focal liver lesions rated on a 9-point scale. We modeled the readers' scores as bimodal distributions in additive Gaussian noise and estimated the distribution parameters from the scores using an expectation maximization algorithm. We (a) sampled 171 similarity scores to simulate a ground truth and (b) simulated readers by adding noise, with standard deviation between 0 and 5 for each reader. We computed the mean values of 2-50 readers' scores and calculated the agreement (AGT) between these means and the simulated ground truth, and the inter-reader agreement (IRA), using Cohen's Kappa metric.
  
  
  RESULTS
  IRA for the empirical data ranged from =0.41 to 0.66. For between 1.5 and 2.5, IRA between three simulated readers was comparable to agreement in the empirical data. For these values , AGT ranged from =0.81 to 0.91. As expected, AGT increased with N, ranging from =0.83 to 0.92 for N = 2 to 50, respectively, with =2.
  
  
  CONCLUSIONS
  Our simulations demonstrated that for moderate to good IRA, excellent AGT could nonetheless be obtained. This model may be used to predict the required N to accurately evaluate similarity in arbitrary size datasets.
authors.affiliation: Electrical Engineering Department, Stanford University, James H. Clark Center, 318 Campus Drive S-324, Stanford, CA 94305, USA. jesscaf@stanford.edu
authors.forename: Jessica
authors.initials: J
authors.lastname: Faruque
authors.forename: Daniel L
authors.initials: DL
authors.lastname: Rubin
authors.forename: Christopher F
authors.initials: CF
authors.lastname: Beaulieu
authors.forename: Sandy
authors.initials: S
authors.lastname: Napel
mesh_terms: D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D000465:Algorithms; D016477:Artifacts; D005260:Female; D006801:Humans; D008099:Liver / Q000000981:diagnostic imaging; D008113:Liver Neoplasms / Q000000981:diagnostic imaging*; D008297:Male; D008875:Middle Aged; D015233:Models, Statistical; D015588:Observer Variation; D015203:Reproducibility of Results; D014057:Tomography, X-Ray Computed / Q000379:methods* / Q000706:statistics & numerical data*