title: Sequential Tests of Multiple Hypotheses Controlling Type I and II Familywise Error Rates.
abstract: This paper addresses the following general scenario: A scientist wishes to perform a battery of experiments, each generating a sequential stream of data, to investigate some phenomenon. The scientist would like to control the overall error rate in order to draw statistically-valid conclusions from each experiment, while being as efficient as possible. The between-stream data may differ in distribution and dimension but also may be highly correlated, even duplicated exactly in some cases. Treating each experiment as a hypothesis test and adopting the familywise error rate (FWER) metric, we give a procedure that sequentially tests each hypothesis while controlling both the type I and II FWERs regardless of the between-stream correlation, and only requires arbitrary sequential test statistics that control the error rates for a given stream in isolation. The proposed procedure, which we call the sequential Holm procedure because of its inspiration from Holm's (1979) seminal fixed-sample procedure, shows simultaneous savings in expected sample size and less conservative error control relative to fixed sample, sequential Bonferroni, and other recently proposed sequential procedures in a simulation study.
authors.affiliation: Department of Mathematics, University of Southern California, Los Angeles, California, USA.
authors.forename: Jay
authors.initials: J
authors.lastname: Bartroff
authors.affiliation: Analysis Group, 111 Huntington Avenue, Tenth Floor, Boston, MA 02199, USA.
authors.forename: Jinlin
authors.initials: J
authors.lastname: Song
keywords: Clinical trials; Holm’s procedure; Multiple testing; Sequential analysis; Step-down test; Wald approximations
references.citation: Stat Med. 2013 Mar 30;32(7):1112-24
references.pmid: 23239078
references.citation: Control Clin Trials. 1998 Dec;19(6):604-21
references.pmid: 9875839
references.citation: JAMA. 2002 Jul 17;288(3):321-33
references.pmid: 12117397
references.citation: Biometrics. 1984 Dec;40(4):1079-87
references.pmid: 6534410
references.citation: JAMA. 2004 Apr 14;291(14):1701-12
references.pmid: 15082697
references.citation: Biometrics. 1993 Mar;49(1):23-30
references.pmid: 8513104
references.citation: Seq Anal. 2020;39(1):65-91
references.pmid: 33776197
references.citation: Biometrics. 1994 Dec;50(4):1146-52
references.pmid: 7786995
references.citation: Stat Med. 2008 May 10;27(10):1593-611
references.pmid: 18275090
references.citation: Biometrics. 1993 Sep;49(3):741-52
references.pmid: 8241370
references.citation: Biometrics. 1979 Sep;35(3):549-56
references.pmid: 497341
references.citation: JAMA. 2004 Jun 23;291(24):2959-68
references.pmid: 15213207