title: Validation of Diagnostic Groups Based on Health Care Utilization Data Should Adjust for Sampling Strategy.
abstract: Valid measurement of outcomes such as disease prevalence using health care utilization data is fundamental to the implementation of a "learning health system." Definitions of such outcomes can be complex, based on multiple diagnostic codes. The literature on validating such data demonstrates a lack of awareness of the need for a stratified sampling design and corresponding statistical methods. We propose a method for validating the measurement of diagnostic groups that have: (1) different prevalences of diagnostic codes within the group; and (2) low prevalence.
  
  
  
  We describe an estimation method whereby: (1) low-prevalence diagnostic codes are oversampled, and the positive predictive value (PPV) of the diagnostic group is estimated as a weighted average of the PPV of each diagnostic code; and (2) claims that fall within a low-prevalence diagnostic group are oversampled relative to claims that are not, and bias-adjusted estimators of sensitivity and specificity are generated.
  
  
  
  We illustrate our proposed method using an example from population health surveillance in which diagnostic groups are applied to physician claims to identify cases of acute respiratory illness.
  
  
  
  Failure to account for the prevalence of each diagnostic code within a diagnostic group leads to the underestimation of the PPV, because low-prevalence diagnostic codes are more likely to be false positives. Failure to adjust for oversampling of claims that fall within the low-prevalence diagnostic group relative to those that do not leads to the overestimation of sensitivity and underestimation of specificity.
authors.affiliation: *Dalla Lana School of Public Health, University of Toronto, Toronto, ON †Department of Epidemiology, Biostatistics and Occupational Health, McGill University ‡Direction de la Santé Publique de Montréal §Department of Medicine, McGill University, Montreal, QC, Canada.
authors.forename: Geneviève
authors.initials: G
authors.lastname: Cadieux
authors.forename: Robyn
authors.initials: R
authors.lastname: Tamblyn
authors.forename: David L
authors.initials: DL
authors.lastname: Buckeridge
authors.forename: Nandini
authors.initials: N
authors.lastname: Dendukuri
mesh_terms: D000465:Algorithms; D003953:Diagnosis-Related Groups; D006801:Humans; D038801:International Classification of Diseases; D010342:Patient Acceptance of Health Care; D011159:Population Surveillance / Q000379:methods*; D012042:Registries; D015983:Selection Bias
references.citation: J Clin Epidemiol. 2000 Apr;53(4):343-9
references.pmid: 10785564
references.citation: Am J Epidemiol. 1997 Feb 1;145(3):227-33
references.pmid: 9012595
references.citation: J Am Med Inform Assoc. 2005 Nov-Dec;12(6):618-29
references.pmid: 16049227
references.citation: J Clin Epidemiol. 1997 Jun;50(6):711-8
references.pmid: 9250269
references.citation: J Hosp Infect. 2008 Jun;69(2):164-8
references.pmid: 18448199
references.citation: J Clin Epidemiol. 2004 Feb;57(2):131-41
references.pmid: 15125622
references.citation: BMC Public Health. 2011 Jan 07;11:17
references.pmid: 21211054
references.citation: Inj Prev. 2005 Jun;11(3):186-90
references.pmid: 15933413
references.citation: J Clin Epidemiol. 1994 Sep;47(9):1027-32
references.pmid: 7730905
references.citation: Am J Epidemiol. 1994 Oct 15;140(8):759-69
references.pmid: 7942777
references.citation: Health Serv Res. 1991 Apr;26(1):53-74
references.pmid: 1901841
references.citation: Annu Rev Public Health. 2001;22:213-30
references.pmid: 11274519
references.citation: Am J Med Qual. 2006 Jul-Aug;21(4):238-45
references.pmid: 16849780
references.citation: J Am Med Inform Assoc. 2009 May-Jun;16(3):354-61
references.pmid: 19261941
references.citation: BMC Med Inform Decis Mak. 2010 Feb 23;10:10
references.pmid: 20178586
references.citation: Med Care. 1994 Jul;32(7 Suppl):JS38-51
references.pmid: 8028412
references.citation: Ann Intern Med. 1993 Oct 15;119(8):844-50
references.pmid: 8018127
references.citation: Mil Med. 2007 Apr;172(4):346-52
references.pmid: 17484301
references.citation: Stroke. 2002 Oct;33(10):2465-70
references.pmid: 12364739
references.citation: Health Serv Res. 2008 Feb;43(1 Pt 1):313-26
references.pmid: 18211532
references.citation: J Clin Epidemiol. 2000 Jun;53(6):605-14
references.pmid: 10880779
references.citation: J Clin Epidemiol. 2003 Jun;56(6):515-9
references.pmid: 12873645
references.citation: Med Care. 2002 Aug;40(8):675-85
references.pmid: 12187181
references.citation: Can J Public Health. 2009 Jul-Aug;100(4):253-7
references.pmid: 19722336
references.citation: Med Care. 2005 May;43(5):480-5
references.pmid: 15838413
references.citation: J Clin Epidemiol. 1992 Jun;45(6):613-9
references.pmid: 1607900
references.citation: J Clin Epidemiol. 2010 Dec;63(12):1332-41
references.pmid: 20457509
references.citation: Biometrics. 1983 Mar;39(1):207-15
references.pmid: 6871349
references.citation: N Engl J Med. 2010 Aug 5;363(6):501-4
references.pmid: 20647183
references.citation: Aust N Z J Public Health. 1997 Aug;21(5):477-82
references.pmid: 9343891
references.citation: J Clin Epidemiol. 2008 Dec;61(12):1250-60
references.pmid: 18619800
references.citation: Acad Med. 2014 Apr;89(4):564-72
references.pmid: 24556775
references.citation: J Am Coll Surg. 2002 Mar;194(3):257-66
references.pmid: 11893128