title: Use of outcomes to evaluate surveillance systems for bioterrorist attacks.
abstract: BACKGROUND
  Syndromic surveillance systems can potentially be used to detect a bioterrorist attack earlier than traditional surveillance, by virtue of their near real-time analysis of relevant data. Receiver operator characteristic (ROC) curve analysis using the area under the curve (AUC) as a comparison metric has been recommended as a practical evaluation tool for syndromic surveillance systems, yet traditional ROC curves do not account for timeliness of detection or subsequent time-dependent health outcomes.
  
  
  METHODS
  Using a decision-analytic approach, we predicted outcomes, measured in lives, quality adjusted life years (QALYs), and costs, for a series of simulated bioterrorist attacks. We then evaluated seven detection algorithms applied to syndromic surveillance data using outcomes-weighted ROC curves compared to simple ROC curves and timeliness-weighted ROC curves. We performed sensitivity analyses by varying the model inputs between best and worst case scenarios and by applying different methods of AUC calculation.
  
  
  RESULTS
  The decision analytic model results indicate that if a surveillance system was successful in detecting an attack, and measures were immediately taken to deliver treatment to the population, the lives, QALYs and dollars lost could be reduced considerably. The ROC curve analysis shows that the incorporation of outcomes into the evaluation metric has an important effect on the apparent performance of the surveillance systems. The relative order of performance is also heavily dependent on the choice of AUC calculation method.
  
  
  CONCLUSIONS
  This study demonstrates the importance of accounting for mortality, morbidity and costs in the evaluation of syndromic surveillance systems. Incorporating these outcomes into the ROC curve analysis allows for more accurate identification of the optimal method for signaling a possible bioterrorist attack. In addition, the parameters used to construct an ROC curve should be given careful consideration.
authors.affiliation: Harvard School of Public Health, Boston, Massachusetts, USA. kerrymcbrien@gmail.com
authors.forename: Kerry A
authors.initials: KA
authors.lastname: McBrien
authors.forename: Ken P
authors.initials: KP
authors.lastname: Kleinman
authors.forename: Allyson M
authors.initials: AM
authors.lastname: Abrams
authors.forename: Lisa A
authors.initials: LA
authors.lastname: Prosser
mesh_terms: D000465:Algorithms; D019540:Area Under Curve; D023482:Bioterrorism; D003198:Computer Simulation / Q000191:economics; D003365:Costs and Cost Analysis; D003661:Decision Support Techniques; D005189:False Positive Reactions; D005260:Female; D006801:Humans; D008297:Male; D017063:Outcome Assessment, Health Care; D011159:Population Surveillance / Q000379:methods*; D019057:Quality-Adjusted Life Years; D012372:ROC Curve; D012680:Sensitivity and Specificity
references.citation: Ann Intern Med. 2006 Feb 21;144(4):270-80
references.pmid: 16490913
references.citation: MMWR Suppl. 2005 Aug 26;54:101-8
references.pmid: 16177700
references.citation: J Biomed Inform. 2005 Apr;38(2):99-113
references.pmid: 15797000
references.citation: JAMA. 2004 Apr 28;291(16):1994-8
references.pmid: 15113818
references.citation: Ann Emerg Med. 2004 Mar;43(3):318-28
references.pmid: 14985657
references.citation: Annu Rev Public Health. 2000;21:587-611
references.pmid: 10884966
references.citation: Pharmacoeconomics. 1998 Apr;13(4):421-33
references.pmid: 10178666
references.citation: MMWR Suppl. 2004 Sep 24;53:5-11
references.pmid: 15714620
references.citation: Epidemiol Infect. 2005 Jun;133(3):409-19
references.pmid: 15962547
references.citation: Emerg Infect Dis. 2001 Nov-Dec;7(6):933-44
references.pmid: 11747719
references.citation: MMWR Morb Mortal Wkly Rep. 2001 Nov 16;50(45):1008-10
references.pmid: 11724158
references.citation: Natl Vital Stat Rep. 2004 Nov 10;53(6):1-38
references.pmid: 15580947
references.citation: Emerg Infect Dis. 2002 Oct;8(10):1124-32
references.pmid: 12396927
references.citation: Health Serv Res. 1988 Jun;23(2):203-35
references.pmid: 3384669
references.citation: Stat Med. 2008 Sep 10;27(20):4057-68
references.pmid: 18407576
references.citation: Stat Methods Med Res. 2006 Oct;15(5):445-64
references.pmid: 17089948
references.citation: J Clin Pathol. 2003 Mar;56(3):182-7
references.pmid: 12610093