title: Testing for ontological errors in probabilistic forecasting models of natural systems.
abstract: Probabilistic forecasting models describe the aleatory variability of natural systems as well as our epistemic uncertainty about how the systems work. Testing a model against observations exposes ontological errors in the representation of a system and its uncertainties. We clarify several conceptual issues regarding the testing of probabilistic forecasting models for ontological errors: the ambiguity of the aleatory/epistemic dichotomy, the quantification of uncertainties as degrees of belief, the interplay between Bayesian and frequentist methods, and the scientific pathway for capturing predictability. We show that testability of the ontological null hypothesis derives from an experimental concept, external to the model, that identifies collections of data, observed and not yet observed, that are judged to be exchangeable when conditioned on a set of explanatory variables. These conditional exchangeability judgments specify observations with well-defined frequencies. Any model predicting these behaviors can thus be tested for ontological error by frequentist methods; e.g., using P values. In the forecasting problem, prior predictive model checking, rather than posterior predictive checking, is desirable because it provides more severe tests. We illustrate experimental concepts using examples from probabilistic seismic hazard analysis. Severe testing of a model under an appropriate set of experimental concepts is the key to model validation, in which we seek to know whether a model replicates the data-generating process well enough to be sufficiently reliable for some useful purpose, such as long-term seismic forecasting. Pessimistic views of system predictability fail to recognize the power of this methodology in separating predictable behaviors from those that are not.
authors.affiliation: Centro per la Pericolosità Sismica, Istituto Nazionale di Geofisica e Vulcanologia, 00143 Rome, Italy; and warner.marzocchi@ingv.it tjordan@usc.edu.
authors.forename: Warner
authors.initials: W
authors.lastname: Marzocchi
authors.affiliation: Southern California Earthquake Center, Department of Earth Sciences, University of Southern California, Los Angeles, CA 90089 warner.marzocchi@ingv.it tjordan@usc.edu.
authors.forename: Thomas H
authors.initials: TH
authors.lastname: Jordan
keywords: Bayesian statistics; expert opinion; significance testing; subjective probability; system science
references.citation: Nature. 2011 Apr 28;472(7344):407-9
references.pmid: 21490595
references.citation: Risk Anal. 2012 Oct;32(10):1607-29
references.pmid: 22489541
references.citation: Br J Math Stat Psychol. 2013 Feb;66(1):8-38
references.pmid: 22364575
references.citation: Br J Math Stat Psychol. 2013 Feb;66(1):68-75
references.pmid: 23017054
references.citation: Science. 1994 Feb 4;263(5147):641-6
references.pmid: 17747657