title: A human judgment approach to epidemiological forecasting.
abstract: Infectious diseases impose considerable burden on society, despite significant advances in technology and medicine over the past century. Advanced warning can be helpful in mitigating and preparing for an impending or ongoing epidemic. Historically, such a capability has lagged for many reasons, including in particular the uncertainty in the current state of the system and in the understanding of the processes that drive epidemic trajectories. Presently we have access to data, models, and computational resources that enable the development of epidemiological forecasting systems. Indeed, several recent challenges hosted by the U.S. government have fostered an open and collaborative environment for the development of these technologies. The primary focus of these challenges has been to develop statistical and computational methods for epidemiological forecasting, but here we consider a serious alternative based on collective human judgment. We created the web-based "Epicast" forecasting system which collects and aggregates epidemic predictions made in real-time by human participants, and with these forecasts we ask two questions: how accurate is human judgment, and how do these forecasts compare to their more computational, data-driven alternatives? To address the former, we assess by a variety of metrics how accurately humans are able to predict influenza and chikungunya trajectories. As for the latter, we show that real-time, combined human predictions of the 2014-2015 and 2015-2016 U.S. flu seasons are often more accurate than the same predictions made by several statistical systems, especially for short-term targets. We conclude that there is valuable predictive power in collective human judgment, and we discuss the benefits and drawbacks of this approach.
authors.affiliation: School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
authors.forename: David C
authors.initials: DC
authors.lastname: Farrow
authors.affiliation: School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Logan C
authors.initials: LC
authors.lastname: Brooks
authors.affiliation: Department of Statistics, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Sangwon
authors.identifier: 0000-0003-0377-897X
authors.initials: S
authors.lastname: Hyun
authors.affiliation: Department of Statistics, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Ryan J
authors.identifier: 0000-0002-2158-8304
authors.initials: RJ
authors.lastname: Tibshirani
authors.affiliation: Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Donald S
authors.identifier: 0000-0002-5704-8094
authors.initials: DS
authors.lastname: Burke
authors.affiliation: School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Roni
authors.initials: R
authors.lastname: Rosenfeld
mesh_terms: D003141:Communicable Diseases / Q000401:mortality*; D004196:Disease Outbreaks / Q000706:statistics & numerical data*; D004812:Epidemiologic Methods; D005544:Forecasting / Q000379:methods*; D006801:Humans; D015233:Models, Statistical; D015995:Prevalence; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000379:methods*; D012680:Sensitivity and Specificity; D014481:United States / Q000453:epidemiology