title: Some surprising facts about (the problem of) surprising facts (from the Dusseldorf Conference, February 2011).
abstract: A common intuition about evidence is that if data x have been used to construct a hypothesis H, then x should not be used again in support of H. It is no surprise that x fits H, if H was deliberately constructed to accord with x. The question of when and why we should avoid such "double-counting" continues to be debated in philosophy and statistics. It arises as a prohibition against data mining, hunting for significance, tuning on the signal, and ad hoc hypotheses, and as a preference for predesignated hypotheses and "surprising" predictions. I have argued that it is the severity or probativeness of the test--or lack of it--that should determine whether a double-use of data is admissible. I examine a number of surprising ambiguities and unexpected facts that continue to bedevil this debate.
authors.forename: D
authors.initials: D
authors.lastname: Mayo
mesh_terms: D036262:Empirical Research; D005544:Forecasting; D019545:Intuition; D012107:Research Design; D012586:Science