Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations.
OBJECTIVE
Model specification-what adjusting variables are analytically modeled-may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the "vibration of effects" (VoE).


METHODS
We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data. We selected 13 variables as adjustment covariates and computed 8,192 Cox models for each of 417 variables' associations with all-cause mortality.


RESULTS
We present the VoE by assessing the variance of the effect size and in the -log10(P-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and P-values and the trajectory of results with increasing adjustments. For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses. For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for mortality.


CONCLUSIONS
Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.
Department of Biomedical Informatics, Harvard Medical School, 10 Shattuck St., Room 314A, Boston, MA 02115, USA.
Chirag J
CJ
Patel
Melbourne School of Population and Global Health, Level 4, 207 Bouverie St., The University of Melbourne, Victoria 3010, Australia.
Belinda
B
Burford
Department of Biomedical Informatics, Harvard Medical School, 10 Shattuck St., Room 314A, Boston, MA 02115, USA; Department of Medicine, Stanford Prevention Research Center, Stanford University School of Medicine, Medical School Office Building, Room X306, 1265 Welch Rd, Stanford, CA 94305, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA 94305, USA. Electronic address: jioannid@stanford.edu.
John P A
JP
Ioannidis
D015986:Confounding Factors, Epidemiologic; D004781:Environmental Exposure / Q000706:statistics & numerical data; D004812:Epidemiologic Methods; D006801:Humans; D015233:Models, Statistical; D009026:Mortality / Q000639:trends*; D009749:Nutrition Surveys; D064887:Observational Studies as Topic; D010641:Phenotype; D018570:Risk Assessment; D014481:United States / Q000453:epidemiology; D014810:Vitamin E / Q000008:administration & dosage
D014810:Vitamin E
Biostatistics; Confounding; Environment-wide association study; Model specification; Observational association; Vibration of effects