title: Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS).
abstract: BACKGROUND
  Diagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.
  
  
  METHODS
  Breast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman's age (40-49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.
  
  
  CONCLUSIONS
  We found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.
authors: Department of Medicine, University of Washington, Seattle, WA, USA. nvoster@uw.edu
authors: Natalia V
authors: NV
authors: Oster
authors: Patricia A
authors: PA
authors: Carney
authors: Kimberly H
authors: KH
authors: Allison
authors: Donald L
authors: DL
authors: Weaver
authors: Lisa M
authors: LM
authors: Reisch
authors: Gary
authors: G
authors: Longton
authors: Tracy
authors: T
authors: Onega
authors: Margaret
authors: M
authors: Pepe
authors: Berta M
authors: BM
authors: Geller
authors: Heidi D
authors: HD
authors: Nelson
authors: Tyler R
authors: TR
authors: Ross
authors: Aanna N A
authors: AN
authors: Tosteson
authors: Joann G
authors: JG
authors: Elmore
mesh_terms: D001940:Breast / Q000473:pathology; D001941:Breast Diseases / Q000473:pathology*; D001943:Breast Neoplasms / Q000145:classification* / Q000175:diagnosis / Q000473:pathology*; D003937:Diagnosis, Differential; D005260:Female; D006801:Humans; D015588:Observer Variation; D011237:Predictive Value of Tests; D011785:Quality Assurance, Health Care / Q000592:standards*; D015203:Reproducibility of Results; D012107:Research Design / Q000592:standards; D012680:Sensitivity and Specificity
references: Int J Cardiol. 2010 Feb 4;138(3):308-10
references: 18762344
references: Am J Surg Pathol. 2004 Jan;28(1):126-31
references: 14707874
references: Neurology. 2006 Sep 12;67(5):792-7
references: 16966539
references: Am J Epidemiol. 2000 Aug 15;152(4):371-8
references: 10968382
references: Br J Cancer. 2008 Nov 4;99(9):1369-74
references: 18781174
references: Radiology. 2010 Oct;257(1):14-7
references: 20851935
references: Med Decis Making. 1997 Jan-Mar;17(1):71-9
references: 8994153
references: Radiology. 2003 Aug;228(2):303-8
references: 12819342
references: Am J Surg Pathol. 1992 Dec;16(12):1133-43
references: 1463092
references: Hum Pathol. 2010 Jun;41(6):895-901
references: 20233620
references: BMJ. 2004 Oct 16;329(7471):883
references: 15469946
references: Am J Surg Pathol. 1991 Mar;15(3):209-21
references: 1847606
references: AJR Am J Roentgenol. 1997 Oct;169(4):1001-8
references: 9308451
references: Ann Surg Oncol. 2005 Aug;12(8):660-73
references: 15968496
references: Radiol Med. 2011 Apr;116(3):466-76
references: 21225368
references: J Natl Cancer Inst. 1998 Jan 21;90(2):142-5
references: 9450574
references: BMC Health Serv Res. 2002;2:4
references: 11884248
references: AJR Am J Roentgenol. 1996 Aug;167(2):367-72
references: 8686606
references: JAMA. 1996 Dec 4;276(21):1752-5
references: 8940325
references: Ann Clin Lab Sci. 2009 Summer;39(3):270-6
references: 19667411
references: Cancer. 2006 Feb 15;106(4):732-42
references: 16411214
references: Radiology. 2005 May;235(2):347-53
references: 15770041
references: N Engl J Med. 1994 Dec 1;331(22):1493-9
references: 7969300
references: Radiology. 2008 Oct;249(1):47-53
references: 18682584
references: Ann Intern Med. 2003 Jan 7;138(1):W1-12
references: 12513067
references: J Clin Epidemiol. 2011 Jan;64(1):96-106
references: 21130355
references: Magn Reson Med Sci. 2010;9(4):217-25
references: 21187691