title: Reducing overconfidence in the interval judgments of experts.
abstract: Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)-a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18]).
authors.affiliation: School of Psychological Science, La Trobe University, Australia.
authors.forename: Andrew
authors.initials: A
authors.lastname: Speirs-Bridge
authors.forename: Fiona
authors.initials: F
authors.lastname: Fidler
authors.forename: Marissa
authors.initials: M
authors.lastname: McBride
authors.forename: Louisa
authors.initials: L
authors.lastname: Flander
authors.forename: Geoff
authors.initials: G
authors.lastname: Cumming
authors.forename: Mark
authors.initials: M
authors.lastname: Burgman
mesh_terms: D016001:Confidence Intervals; D006801:Humans; D007600:Judgment; D011634:Public Health; D018570:Risk Assessment