title: Quantifying the determinants of outbreak detection performance through simulation and machine learning.
abstract: OBJECTIVE
  To develop a probabilistic model for discovering and quantifying determinants of outbreak detection and to use the model to predict detection performance for new outbreaks.
  
  
  METHODS
  We used an existing software platform to simulate waterborne disease outbreaks of varying duration and magnitude. The simulated data were overlaid on real data from visits to emergency department in Montreal for gastroenteritis. We analyzed the combined data using biosurveillance algorithms, varying their parameters over a wide range. We then applied structure and parameter learning algorithms to the resulting data set to build a Bayesian network model for predicting detection performance as a function of outbreak characteristics and surveillance system parameters. We evaluated the predictions of this model through 5-fold cross-validation.
  
  
  RESULTS
  The model predicted performance metrics of commonly used outbreak detection methods with an accuracy greater than 0.80. The model also quantified the influence of different outbreak characteristics and parameters of biosurveillance algorithms on detection performance in practically relevant surveillance scenarios. In addition to identifying characteristics expected a priori to have a strong influence on detection performance, such as the alerting threshold and the peak size of the outbreak, the model suggested an important role for other algorithm features, such as adjustment for weekly patterns.
  
  
  CONCLUSIONS
  We developed a model that accurately predicts how characteristics of disease outbreaks and detection methods will influence on detection. This model can be used to compare the performance of detection methods under different surveillance scenarios, to gain insight into which characteristics of outbreaks and biosurveillance algorithms drive detection performance, and to guide the configuration of surveillance systems.
authors.affiliation: Department of Computer Engineering, Ecole Polytechnique de Montreal, C.P. 6079, succursale Centre-ville, Montreal, Quebec H3C 3A7, Canada. Electronic address: nastaran.jafarpour@polymtl.ca.
authors.forename: Nastaran
authors.initials: N
authors.lastname: Jafarpour
authors.affiliation: Department of Epidemiology and Biostatistics, McGill University, Clinical and Health Informatics Research Group, 1140 Pine Ave. West, Montreal, Quebec H3A 1A3, Canada. Electronic address: mtabae@cs.mcgill.ca.
authors.forename: Masoumeh
authors.initials: M
authors.lastname: Izadi
authors.affiliation: School of Computer Science, McGill University, 3480 University St., Montreal, Quebec H3A 0E7, Canada. Electronic address: dprecup@cs.mcgill.ca.
authors.forename: Doina
authors.initials: D
authors.lastname: Precup
authors.affiliation: Department of Epidemiology and Biostatistics, McGill University, Clinical and Health Informatics Research Group, 1140 Pine Ave. West, Montreal, Quebec H3A 1A3, Canada. Electronic address: david.buckeridge@mcgill.ca.
authors.forename: David L
authors.initials: DL
authors.lastname: Buckeridge
mesh_terms: D000465:Algorithms; D001499:Bayes Theorem; D019295:Computational Biology; D003198:Computer Simulation; D004196:Disease Outbreaks; D005189:False Positive Reactions; D006801:Humans; D000069550:Machine Learning; D011336:Probability; D012372:ROC Curve; D012680:Sensitivity and Specificity
keywords: Bayesian networks; Disease outbreak detection; Outbreak simulation; Predicting performance; Public health informatics; Surveillance