title: Evaluation of radiologist interpretive performance using blinded reads by multiple external readers.
abstract: OBJECTIVE
  To demonstrate a method of assessing radiologic diagnostic agreement utilizing multiple blinded external readers.
  
  
  METHODS
  Six body CT studies interpreted by one reader (primary reader) at the host institution were compiled with patient identifiers removed. Brief clinical histories that were available to the primary reader were provided. Radiologists at 22 centers participated and the interpretations were analyzed in aggregate with the consensus majority that served as the surrogate gold standard for each case.
  
  
  RESULTS
  A total of 31 radiologists formed the group of secondary readers with two-thirds in academic practice averaging 8 years of experience (range: 1-25 years). The average findings per reader for cases A to F include: 1.9 (range: 1-5), 6.3 (range: 2-10), 10.4 (range: 7-14), 5.7 (range: 3-10), 4.2 (range: 2-8), and 3.8 (range: 1-7), respectively. There was agreement of the primary interpretation and the surrogate gold standard for each case.
  
  
  CONCLUSIONS
  The results of our study demonstrate a wide range of interpretation, with wider ranges observed in more complex cases and with vague clinical complaints. Comparison to the primary reader required the use of aggregate analysis and an agreement percentage cutoff to minimize bias and the limitations of this type of study. An intensive evaluation of radiologist performance such as this could be considered in various settings such as a quality assurance program, intense scrutiny of an individual radiologist whether competency is in question, or for medicolegal purpose to ascertain standard of care.
authors.affiliation: Department of Radiology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599-7510, USA.
authors.forename: Andrew F
authors.initials: AF
authors.lastname: Ryan
authors.forename: Richard C
authors.initials: RC
authors.lastname: Semelka
authors.forename: Paul L
authors.initials: PL
authors.lastname: Molina
authors.forename: Shannon
authors.initials: S
authors.lastname: Yonkers
authors.forename: Georgeta
authors.initials: G
authors.lastname: Vaidean
mesh_terms: D002983:Clinical Competence / Q000706:statistics & numerical data*; D003937:Diagnosis, Differential; D005240:Feasibility Studies; D006801:Humans; D015588:Observer Variation; D011785:Quality Assurance, Health Care / Q000379:methods* / Q000706:statistics & numerical data; D011871:Radiology / Q000379:methods / Q000592:standards* / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D014057:Tomography, X-Ray Computed / Q000379:methods / Q000592:standards* / Q000706:statistics & numerical data