title: A New Technique for Evaluating Land-use Regression Models and Their Impact on Health Effect Estimates.
abstract: BACKGROUND
  Leave-one-out cross-validation that fails to account for variable selection does not properly reflect prediction accuracy when the number of training sites is small. The impact on health effect estimates has rarely been studied. The objective of this study was to develop an improved validation procedure for land-use regression models with variable selection and investigate health effect estimates in relation to land-use regression model performance.
  
  
  METHODS
  We randomly generated 10 training and test sets for nitrogen dioxide and particulate matter. For each training set, we developed models and evaluated them using a cross-holdout validation approach. Cross-holdout validation develops new models for each evaluation compared with refitting the model without variable selection, as in standard leave-one-out cross-validation. We also implemented holdout validation, which evaluates model predictions using independent test sets. We evaluated the relationship between cross-holdout validation and holdout validation R and estimates of the association between air pollution and forced vital capacity in the Dutch birth cohort.
  
  
  RESULTS
  Cross-holdout validation Rs were generally identical to holdout validation Rs, but were notably smaller than leave-one-out cross-validation Rs. Decreases in forced vital capacity in relation to air pollution exposure were larger for land-use regression models that had larger holdout validation and cross-holdout validation Rs rather than leave-one-out cross-validation R.
  
  
  CONCLUSIONS
  Cross-holdout validation accurately reflects predictive ability of land-use regression models and is a useful validation approach for small datasets. Land-use regression predictive ability in terms of holdout validation and cross-holdout validation rather than leave-one-out cross-validation was associated with the magnitude of health effect estimates in a case study.
authors.affiliation: From the aInstitute for Risk Assessment Sciences, Utrecht University, Utrecht, The Netherlands; bDepartment of Environmental and Occupational Health Sciences, University of Washington, Seattle, WA; cJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands; and dDepartment of Biostatistics, University of Washington, Seattle, WA.
authors.forename: Meng
authors.initials: M
authors.lastname: Wang
authors.forename: Bert
authors.initials: B
authors.lastname: Brunekreef
authors.forename: Ulrike
authors.initials: U
authors.lastname: Gehring
authors.forename: Adam
authors.initials: A
authors.lastname: Szpiro
authors.forename: Gerard
authors.initials: G
authors.lastname: Hoek
authors.forename: Rob
authors.initials: R
authors.lastname: Beelen
mesh_terms: D000393:Air Pollutants / Q000009:adverse effects / Q000032:analysis*; D000397:Air Pollution / Q000009:adverse effects / Q000032:analysis*; D001530:Belgium; D004781:Environmental Exposure / Q000009:adverse effects / Q000032:analysis*; D004784:Environmental Monitoring / Q000379:methods; D005260:Female; D006801:Humans; D008297:Male; D015233:Models, Statistical; D009426:Netherlands; D009585:Nitrogen Dioxide / Q000009:adverse effects / Q000032:analysis*; D052638:Particulate Matter / Q000009:adverse effects / Q000032:analysis*; D012044:Regression Analysis; D015203:Reproducibility of Results; D014797:Vital Capacity
chemical_list: D000393:Air Pollutants; D052638:Particulate Matter; D009585:Nitrogen Dioxide