Updating meta-analyses leads to larger type I errors than publication bias.
OBJECTIVE
To estimate the extent to which the practice of periodically updating meta-analyses causes inflation of the type I error and then to compare the estimate with the inflation caused by publication bias. We also present a simple method to adjust for the inflation associated with updating meta-analyses.


METHODS
Simulations were used to estimate the error rates.


RESULTS
In general, updating meta-analyses caused 2- to 5-fold inflation of the type I error rates, which exceeded the inflation caused by publication bias. As a rule of thumb, the results of a meta-analysis are robust up to 5, 10, 15, or 22 updates, if the P-value multiplied by 4, 6, 8, or 10 remains below the desired significance level.


CONCLUSIONS
Meta-analyses are likely to be updated until a clear conclusion is reached. Therefore, it is important to take the inflation of the error rate into account to interpret the results correctly.
Borm
George F
GF
Department of Epidemiology, Biostatistics and HTA (EBH 133), Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands. g.borm@ebh.umcn.nl
Donders
A Rogier T
AR
D003627:Data Interpretation, Statistical; D006801:Humans; D015201:Meta-Analysis as Topic; D017594:Publication Bias; D016032:Randomized Controlled Trials as Topic; D016896:Treatment Outcome