title: Practical approach to determine sample size for building logistic prediction models using high-throughput data.
abstract: An empirical method of sample size determination for building prediction models was proposed recently. Permutation method which is used in this procedure is a commonly used method to address the problem of overfitting during cross-validation while evaluating the performance of prediction models constructed from microarray data. But major drawback of such methods which include bootstrapping and full permutations is prohibitively high cost of computation required for calculating the sample size. In this paper, we propose that a single representative null distribution can be used instead of a full permutation by using both simulated and real data sets. During simulation, we have used a dataset with zero effect size and confirmed that the empirical type I error approaches to 0.05. Hence this method can be confidently applied to reduce overfitting problem during cross-validation. We have observed that pilot data set generated by random sampling from real data could be successfully used for sample size determination. We present our results using an experiment that was repeated for 300 times while producing results comparable to that of full permutation method. Since we eliminate full permutation, sample size estimation time is not a function of pilot data size. In our experiment we have observed that this process takes around 30min. With the increasing number of clinical studies, developing efficient sample size determination methods for building prediction models is critical. But empirical methods using bootstrap and permutation usually involve high computing costs. In this study, we propose a method that can reduce required computing time drastically by using representative null distribution of permutations. We use data from pilot experiments to apply this method for designing clinical studies efficiently for high throughput data.
authors.affiliation: Samsung Genome Institute, Samsung Medical Center, Seoul, Republic of Korea; In vitro Diagnostics Research Lab, Bio Research Center, Samsung Advanced Institute of Technology, Gyeonggi-do, Republic of Korea; Department of Statistics, Korea University, Seoul, Republic of Korea. Electronic address: ds3.son@samsung.com.
authors.forename: Dae-Soon
authors.initials: DS
authors.lastname: Son
authors.affiliation: Department of Statistics, Texas A&M University, College Station, TX 77843, USA. Electronic address: dhyuklee@tamu.edu.
authors.forename: DongHyuk
authors.initials: D
authors.lastname: Lee
authors.affiliation: Clinomics, Inc., A-616 Gardenfive Works, Seoul, Republic of Korea. Electronic address: klee@clinomics.co.kr.
authors.forename: Kyusang
authors.initials: K
authors.lastname: Lee
authors.affiliation: Department of Biostatistics and Bioinformatics, Duke University, NC 27710, USA. Electronic address: sinho.jung@duke.edu.
authors.forename: Sin-Ho
authors.initials: SH
authors.lastname: Jung
authors.affiliation: Samsung Genome Institute, Samsung Medical Center, Seoul, Republic of Korea; In vitro Diagnostics Research Lab, Bio Research Center, Samsung Advanced Institute of Technology, Gyeonggi-do, Republic of Korea. Electronic address: taejin.ahn@samsung.com.
authors.forename: Taejin
authors.initials: T
authors.lastname: Ahn
authors.affiliation: Samsung Genome Institute, Samsung Medical Center, Seoul, Republic of Korea; In vitro Diagnostics Research Lab, Bio Research Center, Samsung Advanced Institute of Technology, Gyeonggi-do, Republic of Korea. Electronic address: eunjin.lee@samsung.com.
authors.forename: Eunjin
authors.initials: E
authors.lastname: Lee
authors.affiliation: Samsung Cancer Research Institute, Samsung Medical Center, Seoul, Republic of Korea. Electronic address: insuk.sohn@samsung.com.
authors.forename: Insuk
authors.initials: I
authors.lastname: Sohn
authors.affiliation: Samsung Genome Institute, Samsung Medical Center, Seoul, Republic of Korea; In vitro Diagnostics Research Lab, Bio Research Center, Samsung Advanced Institute of Technology, Gyeonggi-do, Republic of Korea. Electronic address: doogie.chung@samsung.com.
authors.forename: Jongsuk
authors.initials: J
authors.lastname: Chung
authors.affiliation: Samsung Genome Institute, Samsung Medical Center, Seoul, Republic of Korea. Electronic address: woongyang.park@samsung.com.
authors.forename: Woongyang
authors.initials: W
authors.lastname: Park
authors.affiliation: In vitro Diagnostics Research Lab, Bio Research Center, Samsung Advanced Institute of Technology, Gyeonggi-do, Republic of Korea. Electronic address: bio.stat@daum.net.
authors.forename: Nam
authors.initials: N
authors.lastname: Huh
authors.affiliation: Department of Statistics, Korea University, Seoul, Republic of Korea. Electronic address: jael@korea.ac.kr.
authors.forename: Jae Won
authors.initials: JW
authors.lastname: Lee
mesh_terms: D000465:Algorithms; D019295:Computational Biology / Q000379:methods*; D003198:Computer Simulation; D020869:Gene Expression Profiling / Q000379:methods*; D006801:Humans; D016015:Logistic Models; D010865:Pilot Projects; D015203:Reproducibility of Results; D012107:Research Design; D018401:Sample Size; D012984:Software
keywords: Null distribution; Permutation; Prediction and validation; Sample size; Statistical power