title: Measuring interobserver variation in a pathology EQA scheme using weighted κ for multiple readers.
abstract: BACKGROUND
  A Urological Pathology External Quality Assurance (EQA) Scheme in the UK has reported observer variation in the diagnosis and grading of adenocarcinoma in prostatic biopsies using basic κ statistics, which rate all disagreements equally.
  
  
  OBJECTIVE
  The aim of this study is to use customised weighting schemes to report κ statistics that reflect the closeness of interobserver agreement in the prostate EQA scheme.
  
  
  METHODS
  A total of 83, 114 and 116 pathologists took part, respectively, in three web-based circulations and were classified as either expert or other readers. For analyses of diagnosis, there were 10, 8 and 8 cases in the three circulations, respectively. For analyses of Gleason Sum Score, only invasive cases were included, leaving 5, 5 and 6 cases, respectively. Analyses were conducted using customised weighting schemes with 'pairwise-weighted' κ for multiple readers.
  
  
  RESULTS
  Analysis of diagnosis for all circulations and all readers gave a composite κ value of 0.86 and pairwise-weighted κ (κ(p-w)) value of 0.91, both regarded as 'almost perfect' agreement. This was due to the high proportion of responses that showed partial agreement. Analysis of Gleason Sum Score gave κ=0.38 and κ(p-w)=0.58 over all circulations and all readers, indicating that discrepancies occur at the boundary between adjacent grades and may not be as clinically significant as suggested by composite κ.
  
  
  CONCLUSIONS
  Weighted κ show higher levels of agreement than previously reported as they have the advantage of applying weighting, which reflects the relative importance of different types of discordance in diagnosis or grading. Agreement on grading remained low.
authors.affiliation: Cancer Screening Evaluation Unit, Institute of Cancer Research, Sutton, Surrey, UK. karen.wright@iconplc.com
authors.forename: Karen C
authors.initials: KC
authors.lastname: Wright
authors.forename: Jane
authors.initials: J
authors.lastname: Melia
authors.forename: Sue
authors.initials: S
authors.lastname: Moss
authors.forename: Dan M
authors.initials: DM
authors.lastname: Berney
authors.forename: Derek
authors.initials: D
authors.lastname: Coleman
authors.forename: Patricia
authors.initials: P
authors.lastname: Harnden
mesh_terms: D000230:Adenocarcinoma / Q000473:pathology*; D001706:Biopsy / Q000592:standards / Q000706:statistics & numerical data; D006801:Humans; D008297:Male; D060787:Neoplasm Grading / Q000592:standards / Q000706:statistics & numerical data; D015588:Observer Variation; D010338:Pathology, Clinical / Q000592:standards*; D011467:Prostate / Q000473:pathology*; D011471:Prostatic Neoplasms / Q000473:pathology*; D011785:Quality Assurance, Health Care