title: Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases.
abstract: For many societally important science-based decisions, data are inadequate, unreliable or non-existent, and expert advice is sought. In such cases, procedures for eliciting structured expert judgments (SEJ) are increasingly used. This raises questions regarding validity and reproducibility. This paper presents new findings from a large-scale international SEJ study intended to estimate the global burden of foodborne disease on behalf of WHO. The study involved 72 experts distributed over 134 expert panels, with panels comprising thirteen experts on average. Elicitations were conducted in five languages. Performance-based weighted solutions for target questions of interest were formed for each panel. These weights were based on individual expert's statistical accuracy and informativeness, determined using between ten and fifteen calibration variables from the experts' field with known values. Equal weights combinations were also calculated. The main conclusions on expert performance are: (1) SEJ does provide a science-based method for attribution of the global burden of foodborne diseases; (2) equal weighting of experts per panel increased statistical accuracy to acceptable levels, but at the cost of informativeness; (3) performance-based weighting increased informativeness, while retaining accuracy; (4) due to study constraints individual experts' accuracies were generally lower than in other SEJ studies, and (5) there was a negative correlation between experts' informativeness and statistical accuracy which attenuated as accuracy improved, revealing that the least accurate experts drive the negative correlation. It is shown, however, that performance-based weighting has the ability to yield statistically accurate and informative combinations of experts' judgments, thereby offsetting this contrary influence. The present findings suggest that application of SEJ on a large scale is feasible, and motivate the development of enhanced training and tools for remote elicitation of multiple, internationally-dispersed panels.
authors.affiliation: Aspinall & Associates, Tisbury, England.|Bristol University, Bristol, England.
authors.forename: W P
authors.initials: WP
authors.lastname: Aspinall
authors.affiliation: Resources for the Future, Washington D. C., United States of America.|Delft University of Technology, Delft, The Netherlands.
authors.forename: R M
authors.initials: RM
authors.lastname: Cooke
authors.affiliation: National Institute for Public Health and the Environment, Bilthoven, The Netherlands.|University of Florida, Gainesville, Florida, United States of America.|Utrecht University, Utrecht, The Netherlands.
authors.forename: A H
authors.initials: AH
authors.lastname: Havelaar
authors.affiliation: U. S. Dept. of Agriculture, Economic Research Service, Washington D. C., United States of America.
authors.forename: S
authors.initials: S
authors.lastname: Hoffmann
authors.affiliation: Technical University of Denmark, Lyngby, Denmark.
authors.forename: T
authors.initials: T
authors.lastname: Hald
mesh_terms: D005104:Expert Testimony / Q000379:methods*; D005511:Food Handling / Q000379:methods*; D059022:Food Safety / Q000379:methods*; D005517:Foodborne Diseases / Q000175:diagnosis / Q000517:prevention & control*; D014943:Global Health; D006801:Humans; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000379:methods; D012307:Risk Factors; D014944:World Health Organization