title: On stability issues in deriving multivariable regression models.
abstract: In many areas of science where empirical data are analyzed, a task is often to identify important variables with influence on an outcome. Most often this is done by using a variable selection strategy in the context of a multivariable regression model. Using a study on ozone effects in children (n = 496, 24 covariates), we will discuss aspects relevant for deriving a suitable model. With an emphasis on model stability, we will explore and illustrate differences between predictive models and explanatory models, the key role of stopping criteria, and the value of bootstrap resampling (with and without replacement). Bootstrap resampling will be used to assess variable selection stability, to derive a predictor that incorporates model uncertainty, check for influential points, and visualize the variable selection process. For the latter two tasks we adapt and extend recent approaches, such as stability paths, to serve our purposes. Based on earlier experiences and on results from the example, we will argue for simpler models and that predictions are usually very similar, irrespective of the selection method used. Important differences exist for the corresponding variances, and the model uncertainty concept helps to protect against serious underestimation of the variance of a predictor-derived data dependently. Results of stability investigations illustrate severe difficulties in the task of deriving a suitable explanatory model. It seems possible to identify a small number of variables with an important and probably true influence on the outcome, but too often several variables are included whose selection may be a result of chance or may depend on a small number of observations.
authors.affiliation: Department für Medizinische Biometrie und Medizinische Informatik, Universitätsklinikum Freiburg, Stefan-Meier-Str. 26, 79104, Freiburg, Germany.
authors.forename: Willi
authors.initials: W
authors.lastname: Sauerbrei
authors.affiliation: Department für Medizinische Biometrie und Medizinische Informatik, Universitätsklinikum Freiburg, Stefan-Meier-Str. 26, 79104, Freiburg, Germany.
authors.forename: Anika
authors.initials: A
authors.lastname: Buchholz
authors.affiliation: Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie, Ludwig-Maximilians-Universität München, Marchioninistr. 15, 81377, München, Germany.
authors.forename: Anne-Laure
authors.initials: AL
authors.lastname: Boulesteix
authors.affiliation: Institut für Medizinische Biometrie, Epidemiologie und Informatik, Universitätsmedizin der Johannes-Gutenberg-Universität Mainz, Obere Zahlbacher Straße 69, 55131, Mainz, Germany.
authors.forename: Harald
authors.initials: H
authors.lastname: Binder
mesh_terms: D002648:Child; D006801:Humans; D016013:Likelihood Functions; D015233:Models, Statistical; D015999:Multivariate Analysis; D010126:Ozone / Q000009:adverse effects; D012044:Regression Analysis; D012119:Respiration / Q000187:drug effects; D035501:Uncertainty
chemical_list: D010126:Ozone
keywords: Bootstrap resampling; Influential point; Model stability; Regression model; Variable selection