title: P-values in genomics: apparent precision masks high uncertainty.
abstract: Scientists often interpret P-values as measures of the relative strength of statistical findings. This is common practice in large-scale genomic studies where P-values are used to choose which of numerous hypothesis test results should be pursued in subsequent research. In this study, we examine P-value variability to assess the degree of certainty P-values provide. We develop prediction intervals for the P-value in a replication study given the P-value observed in an initial study. The intervals depend on the initial value of P and the ratio of sample sizes between the initial and replication studies, but not on the underlying effect size or initial sample size. The intervals are valid for most large-sample statistical tests in any context, and can be used in the presence of single or multiple tests. While P-values are highly variable, future P-value variability can be explicitly predicted based on a P-value from an initial study. The relative size of the replication and initial study is an important predictor of the P-value in a subsequent replication study. We provide a handy calculator implementing these results and apply them to a study of Alzheimer's disease and recent findings of the Cross-Disorder Group of the Psychiatric Genomics Consortium. This study suggests that overinterpretation of very significant, but highly variable, P-values is an important factor contributing to the unexpectedly high incidence of non-replication. Formal prediction intervals can also provide realistic interpretations and comparisons of P-values associated with different estimated effect sizes and sample sizes.
authors.affiliation: Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, CA, USA.
authors.forename: L C
authors.initials: LC
authors.lastname: Lazzeroni
authors.affiliation: 1] VA Cooperative Studies Program Palo Alto Coordinating Center, Mountain View, CA, USA [2] Department of Health Research and Policy, Stanford University School of Medicine, CA, USA.
authors.forename: Y
authors.initials: Y
authors.lastname: Lu
authors.affiliation: 1] VA Cooperative Studies Program Palo Alto Coordinating Center, Mountain View, CA, USA [2] Department of Population Health, New York University School of Medicine, New York, NY, USA [3] Department of Environmental Medicine, New York University School of Medicine, New York, NY, USA.
authors.forename: I
authors.initials: I
authors.lastname: Belitskaya-Lévy
mesh_terms: D000544:Alzheimer Disease / Q000235:genetics / Q000473:pathology; D001921:Brain / Q000473:pathology; D023281:Genomics / Q000379:methods* / Q000706:statistics & numerical data*; D006801:Humans; D001523:Mental Disorders / Q000235:genetics; D015233:Models, Statistical; D020641:Polymorphism, Single Nucleotide; D035501:Uncertainty