Use of outcomes to evaluate surveillance systems for bioterrorist attacks.
BACKGROUND
Syndromic surveillance systems can potentially be used to detect a bioterrorist attack earlier than traditional surveillance, by virtue of their near real-time analysis of relevant data. Receiver operator characteristic (ROC) curve analysis using the area under the curve (AUC) as a comparison metric has been recommended as a practical evaluation tool for syndromic surveillance systems, yet traditional ROC curves do not account for timeliness of detection or subsequent time-dependent health outcomes.


METHODS
Using a decision-analytic approach, we predicted outcomes, measured in lives, quality adjusted life years (QALYs), and costs, for a series of simulated bioterrorist attacks. We then evaluated seven detection algorithms applied to syndromic surveillance data using outcomes-weighted ROC curves compared to simple ROC curves and timeliness-weighted ROC curves. We performed sensitivity analyses by varying the model inputs between best and worst case scenarios and by applying different methods of AUC calculation.


RESULTS
The decision analytic model results indicate that if a surveillance system was successful in detecting an attack, and measures were immediately taken to deliver treatment to the population, the lives, QALYs and dollars lost could be reduced considerably. The ROC curve analysis shows that the incorporation of outcomes into the evaluation metric has an important effect on the apparent performance of the surveillance systems. The relative order of performance is also heavily dependent on the choice of AUC calculation method.


CONCLUSIONS
This study demonstrates the importance of accounting for mortality, morbidity and costs in the evaluation of syndromic surveillance systems. Incorporating these outcomes into the ROC curve analysis allows for more accurate identification of the optimal method for signaling a possible bioterrorist attack. In addition, the parameters used to construct an ROC curve should be given careful consideration.
McBrien
Kerry A
KA
Harvard School of Public Health, Boston, Massachusetts, USA. kerrymcbrien@gmail.com
Kleinman
Ken P
KP
Abrams
Allyson M
AM
Prosser
Lisa A
LA
D000465:Algorithms; D019540:Area Under Curve; D023482:Bioterrorism; D003198:Computer Simulation / Q000191:economics; D003365:Costs and Cost Analysis; D003661:Decision Support Techniques; D005189:False Positive Reactions; D005260:Female; D006801:Humans; D008297:Male; D017063:Outcome Assessment, Health Care; D011159:Population Surveillance / Q000379:methods*; D019057:Quality-Adjusted Life Years; D012372:ROC Curve; D012680:Sensitivity and Specificity
Ann Intern Med. 2006 Feb 21;144(4):270-80
16490913
MMWR Suppl. 2005 Aug 26;54:101-8
16177700
J Biomed Inform. 2005 Apr;38(2):99-113
15797000
JAMA. 2004 Apr 28;291(16):1994-8
15113818
Ann Emerg Med. 2004 Mar;43(3):318-28
14985657
Annu Rev Public Health. 2000;21:587-611
10884966
Pharmacoeconomics. 1998 Apr;13(4):421-33
10178666
MMWR Suppl. 2004 Sep 24;53:5-11
15714620
Epidemiol Infect. 2005 Jun;133(3):409-19
15962547
Emerg Infect Dis. 2001 Nov-Dec;7(6):933-44
11747719
MMWR Morb Mortal Wkly Rep. 2001 Nov 16;50(45):1008-10
11724158
Natl Vital Stat Rep. 2004 Nov 10;53(6):1-38
15580947
Emerg Infect Dis. 2002 Oct;8(10):1124-32
12396927
Health Serv Res. 1988 Jun;23(2):203-35
3384669
Stat Med. 2008 Sep 10;27(20):4057-68
18407576
Stat Methods Med Res. 2006 Oct;15(5):445-64
17089948
J Clin Pathol. 2003 Mar;56(3):182-7
12610093