title: Combining experts' judgments: comparison of algorithmic methods using synthetic data.
abstract: Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject-matter experts about the value of a decision-relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal-weight combination (the average of the experts' distributions). We evaluate the properties of five combination methods (equal-weight, best-expert, performance, frequentist, and copula) using simulated expert-judgment data for which we know the process generating the experts' distributions. We examine cases in which two well-calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best-expert approaches perform better and the equal-weight combination method performs worse than the alternative approaches.
authors.affiliation: Center for Risk Analysis, Harvard University, 718 Huntington Ave., Boston, MA 02115, USA. jkh@harvard.edu
authors.forename: James K
authors.initials: JK
authors.lastname: Hammitt
authors.forename: Yifan
authors.initials: Y
authors.lastname: Zhang
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D003657:Decision Making; D005104:Expert Testimony / Q000379:methods*; D006801:Humans