title: Pairwise comparison versus Likert scale for biomedical image assessment.
abstract: OBJECTIVE
  Biomedical imaging research relies heavily on the subjective and semi-quantitative reader analysis of images. Current methods are limited by interreader variability and fixed upper and lower limits. The purpose of this study was to compare the performance of two assessment methods, pairwise comparison and Likert scale, for improved analysis of biomedical images.
  
  
  METHODS
  A set of 10 images with varying degrees of image sharpness was created by digitally blurring a normal clinical chest radiograph. Readers assessed the degree of image sharpness using two different methods: pairwise comparison and a 10-point Likert scale. Reader agreement with actual chest radiograph sharpness was calculated for each method by use of the Lin concordance correlation coefficient (CCC).
  
  
  RESULTS
  Reader accuracy was highest for pairwise comparison (CCC, 1.0) and ranked Likert (CCC, 0.99) scores and lowest for nonranked Likert scores (CCC, 0.83). Accuracy improved slightly when readers repeated their assessments (CCC, 0.87) or had reference images available (CCC, 0.91).
  
  
  CONCLUSIONS
  Pairwise comparison and ranked Likert scores yield more accurate reader assessments than nonranked Likert scores.
authors.affiliation: 1 All authors: Department of Radiology and Biomedical Imaging, University of California, San Francisco, Benioff Children's Hospital, 505 Parnassus Ave, Box 0628, M-396, San Francisco, CA 94143.
authors.forename: Andrew S
authors.initials: AS
authors.lastname: Phelps
authors.forename: David M
authors.initials: DM
authors.lastname: Naeger
authors.forename: Jesse L
authors.initials: JL
authors.lastname: Courtier
authors.forename: Jack W
authors.initials: JW
authors.lastname: Lambert
authors.forename: Peter A
authors.initials: PA
authors.lastname: Marcovici
authors.forename: Javier E
authors.initials: JE
authors.lastname: Villanueva-Meyer
authors.forename: John D
authors.initials: JD
authors.lastname: MacKenzie
mesh_terms: D000465:Algorithms; D003627:Data Interpretation, Statistical; D006801:Humans; D015588:Observer Variation; D011594:Psychometrics / Q000379:methods*; D011857:Radiographic Image Interpretation, Computer-Assisted / Q000379:methods*; D013902:Radiography, Thoracic / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity
keywords: Likert scale; image assessment; pairwise comparison