title: Best-case kappa scores calculated retrospectively from EEG report databases.
abstract: OBJECTIVE
  The most popular metric for interrater reliability in electroencephalography is the kappa (κ) score. κ calculation is laborious, requiring EEG readers to read the same EEG studies. We introduce a method to determine the best-case κ score (κBEST) for measuring interrater reliability between EEG readers, retrospectively.
  
  
  METHODS
  We incorporated 1 year of EEG reports read by four adult EEG readers at our institution. We used SQL queries to determine EEG findings for subsequent analysis. We generated logistic regression models for particular EEG findings, dependent on patient age, location acuity, and EEG reader. We derived a novel measure, the κBEST statistic, from the logistic regression coefficients.
  
  
  RESULTS
  Increasing patient age and location acuity were associated with decreased sleep and increased diffuse abnormalities. For certain findings, EEG readers exhibited the dominant influence, manifesting directly as lower between-reader κBEST scores for certain EEG findings. Within-reader κBEST control scores were higher than between-reader scores, suggesting internal consistency.
  
  
  CONCLUSIONS
  The κBEST metric can measure significant interrater reliability differences between any number of EEG readers and reports, retrospectively, and is generalizable to other domains (e.g., pathology or radiology reporting). We suggest using this metric as a guide or starting point for focused quality control efforts.
authors.affiliation: Department of Neurology, UMDNJ-Robert Wood Johnson Medical School, New Brunswick, New Jersey 08901, USA.
authors.forename: Ahmad
authors.initials: A
authors.lastname: Nizam
authors.forename: Sining
authors.initials: S
authors.lastname: Chen
authors.forename: Stephen
authors.initials: S
authors.lastname: Wong
mesh_terms: D002648:Child; D002675:Child, Preschool; D003627:Data Interpretation, Statistical; D016208:Databases, Factual; D003936:Diagnosis, Computer-Assisted / Q000379:methods; D004569:Electroencephalography / Q000379:methods* / Q000706:statistics & numerical data*; D004827:Epilepsy / Q000175:diagnosis*; D005260:Female; D006801:Humans; D007223:Infant; D007231:Infant, Newborn; D008297:Male; D015588:Observer Variation; D011361:Professional Competence / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity