A global goodness-of-fit test for receiver operating characteristic curve analysis via the bootstrap method.
OBJECTIVE
Medical classification accuracy studies often yield continuous data based on predictive models for treatment outcomes. A popular method for evaluating the performance of diagnostic tests is the receiver operating characteristic (ROC) curve analysis. The main objective was to develop a global statistical hypothesis test for assessing the goodness-of-fit (GOF) for parametric ROC curves via the bootstrap.


METHODS
A simple log (or logit) and a more flexible Box-Cox normality transformations were applied to untransformed or transformed data from two clinical studies to predict complications following percutaneous coronary interventions (PCIs) and for image-guided neurosurgical resection results predicted by tumor volume, respectively. We compared a non-parametric with a parametric binormal estimate of the underlying ROC curve. To construct such a GOF test, we used the non-parametric and parametric areas under the curve (AUCs) as the metrics, with a resulting p value reported.


RESULTS
In the interventional cardiology example, logit and Box-Cox transformations of the predictive probabilities led to satisfactory AUCs (AUC=0.888; p=0.78, and AUC=0.888; p=0.73, respectively), while in the brain tumor resection example, log and Box-Cox transformations of the tumor size also led to satisfactory AUCs (AUC=0.898; p=0.61, and AUC=0.899; p=0.42, respectively). In contrast, significant departures from GOF were observed without applying any transformation prior to assuming a binormal model (AUC=0.766; p=0.004, and AUC=0.831; p=0.03), respectively.


CONCLUSIONS
In both studies the p values suggested that transformations were important to consider before applying any binormal model to estimate the AUC. Our analyses also demonstrated and confirmed the predictive values of different classifiers for determining the interventional complications following PCIs and resection outcomes in image-guided neurosurgery.
Zou
Kelly H
KH
Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, MIT, MA, USA. zou@bwh.harvard.edu
Resnic
Frederic S
FS
Talos
Ion-Florin
IF
Goldberg-Zimring
Daniel
D
Bhagwat
Jui G
JG
Haker
Steven J
SJ
Kikinis
Ron
R
Jolesz
Ferenc A
FA
Ohno-Machado
Lucila
L
D000293:Adolescent; D000328:Adult; D000465:Algorithms; D015906:Angioplasty, Balloon, Coronary / Q000401:mortality*; D001932:Brain Neoplasms / Q000401:mortality* / Q000601:surgery; D002138:Calibration; D003627:Data Interpretation, Statistical; D020000:Decision Support Systems, Clinical; D003936:Diagnosis, Computer-Assisted / Q000379:methods*; D016002:Discriminant Analysis; D005103:Expert Systems; D005260:Female; D006801:Humans; D015994:Incidence; D008297:Male; D008875:Middle Aged; D017063:Outcome Assessment, Health Care / Q000379:methods*; D011379:Prognosis; D012372:ROC Curve; D018570:Risk Assessment / Q000379:methods*; D012307:Risk Factors; D016019:Survival Analysis; D015996:Survival Rate; D014481:United States / Q000453:epidemiology