Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality.
OBJECTIVE
Automated variable selection methods are frequently used to determine the independent predictors of an outcome. The objective of this study was to determine the reproducibility of logistic regression models developed using automated variable selection methods.


METHODS
An initial set of 29 candidate variables were considered for predicting mortality after acute myocardial infarction (AMI). We drew 1,000 bootstrap samples from a dataset consisting of 4,911 patients admitted to hospital with an AMI. Using each bootstrap sample, logistic regression models predicting 30-day mortality were obtained using backward elimination, forward selection, and stepwise selection. The agreement between the different model selection methods and the agreement across the 1,000 bootstrap samples were compared.


RESULTS
Using 1,000 bootstrap samples, backward elimination identified 940 unique models for predicting mortality. Similar results were obtained for forward and stepwise selection. Three variables were identified as independent predictors of mortality among all bootstrap samples. Over half the candidate prognostic variables were identified as independent predictors in less than half of the bootstrap samples.


CONCLUSIONS
Automated variable selection methods result in models that are unstable and not reproducible. The variables selected as independent predictors are sensitive to random fluctuations in the data.
Austin
Peter C
PC
Institute for Clinical Evaluative Sciences, G1 06, 2075 Bayview Avenue, Toronto, Ontario M4N 3M5, Canada. peter.austin@ices.on.ca
Tu
Jack V
JV
D000368:Aged; D001330:Electronic Data Processing; D005260:Female; D006801:Humans; D016015:Logistic Models; D008297:Male; D008875:Middle Aged; D009203:Myocardial Infarction / Q000401:mortality*; D015203:Reproducibility of Results; D015983:Selection Bias