title: On the assessment of the added value of new predictive biomarkers.
abstract: BACKGROUND
  The surge in biomarker development calls for research on statistical evaluation methodology to rigorously assess emerging biomarkers and classification models. Recently, several authors reported the puzzling observation that, in assessing the added value of new biomarkers to existing ones in a logistic regression model, statistical significance of new predictor variables does not necessarily translate into a statistically significant increase in the area under the ROC curve (AUC). Vickers et al. concluded that this inconsistency is because AUC "has vastly inferior statistical properties," i.e., it is extremely conservative. This statement is based on simulations that misuse the DeLong et al. method. Our purpose is to provide a fair comparison of the likelihood ratio (LR) test and the Wald test versus diagnostic accuracy (AUC) tests.
  
  
  CONCLUSIONS
  We present a test to compare ideal AUCs of nested linear discriminant functions via an F test. We compare it with the LR test and the Wald test for the logistic regression model. The null hypotheses of these three tests are equivalent; however, the F test is an exact test whereas the LR test and the Wald test are asymptotic tests. Our simulation shows that the F test has the nominal type I error even with a small sample size. Our results also indicate that the LR test and the Wald test have inflated type I errors when the sample size is small, while the type I error converges to the nominal value asymptotically with increasing sample size as expected. We further show that the DeLong et al. method tests a different hypothesis and has the nominal type I error when it is used within its designed scope. Finally, we summarize the pros and cons of all four methods we consider in this paper.
  
  
  CONCLUSIONS
  We show that there is nothing inherently less powerful or disagreeable about ROC analysis for showing the usefulness of new biomarkers or characterizing the performance of classification models. Each statistical method for assessing biomarkers and classification models has its own strengths and weaknesses. Investigators need to choose methods based on the assessment purpose, the biomarker development phase at which the assessment is being performed, the available patient data, and the validity of assumptions behind the methodologies.
authors.affiliation: Division of Imaging and Applied Mathematics, Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, Food and Drug Administration, 10903 New Hampshire Avenue, Silver Spring, MD 20993, USA. weijie.chen@fda.hhs.gov
authors.forename: Weijie
authors.initials: W
authors.lastname: Chen
authors.forename: Frank W
authors.initials: FW
authors.lastname: Samuelson
authors.forename: Brandon D
authors.initials: BD
authors.lastname: Gallas
authors.forename: Le
authors.initials: L
authors.lastname: Kang
authors.forename: Berkman
authors.initials: B
authors.lastname: Sahiner
authors.forename: Nicholas
authors.initials: N
authors.lastname: Petrick
mesh_terms: D019540:Area Under Curve; D015415:Biomarkers; D006801:Humans; D016013:Likelihood Functions; D016015:Logistic Models; D015233:Models, Statistical; D011237:Predictive Value of Tests
chemical_list: D015415:Biomarkers
references.citation: Stat Med. 2012 Oct 15;31(23):2577-87
references.pmid: 22415937
references.citation: Biometrika. 1948 May;35(Pts 1-2):58-79
references.pmid: 18867413
references.citation: BMC Med Res Methodol. 2011 Jan 28;11:13
references.pmid: 21276237
references.citation: J Natl Cancer Inst. 2008 Oct 15;100(20):1432-8
references.pmid: 18840817
references.citation: Am J Epidemiol. 2011 Aug 1;174(3):364-74
references.pmid: 21673124
references.citation: Med Phys. 2008 Apr;35(4):1559-70
references.pmid: 18491550
references.citation: J Natl Cancer Inst. 2001 Jul 18;93(14):1054-61
references.pmid: 11459866
references.citation: Clin Trials. 2006;3(1):43-56
references.pmid: 16539089
references.citation: Biometrics. 1988 Sep;44(3):837-45
references.pmid: 3203132
references.citation: Stat Med. 2013 Apr 30;32(9):1467-82
references.pmid: 23296397
references.citation: Stat Med. 2011 May 30;30(12):1410-8
references.pmid: 21337594