title: A Note on Comparing the Power of Test Statistics at Low Significance Levels.
abstract: It is an obvious fact that the power of a test statistic is dependent upon the significance (alpha) level at which the test is performed. It is perhaps a less obvious fact that therelativeperformance of two statistics in terms of power is also a function of the alpha level. Through numerous personal discussions, we have noted that even some competent statisticians have the mistaken intuition that relative power comparisons at traditional levels such asα= 0.05 will be roughly similar to relative power comparisons at very low levels, such as the levelα= 5 × 10-8, which is commonly used in genome-wide association studies. In this brief note, we demonstrate that this notion is in fact quite wrong, especially with respect to comparing tests with differing degrees of freedom. In fact, at very low alpha levels the cost of additional degrees of freedom is often comparatively low. Thus we recommend that statisticians exercise caution when interpreting the results of power comparison studies which use alpha levels that will not be used in practice.
authors.affiliation: Case Western Reserve University Cleveland, OH 44106-7281, USA.
authors.forename: Nathan
authors.initials: N
authors.lastname: Morris
authors.affiliation: Case Western Reserve University Cleveland, OH 44106-7281, USA.
authors.forename: Robert
authors.initials: R
authors.lastname: Elston
keywords: Power; Small Significance Levels