Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
OBJECTIVE
Because the results of a meta-analysis are used to formulate the highest level recommendation in clinical practice guidelines, clinicians should be mindful of problems inherent in this technique. Rather than reviewing meta-analysis in abstract, general terms, we believe readers can gain a more concrete understanding of the problems through a detailed examination of one meta-analysis. The meta-analysis on which we focus is that conducted by an American College of Chest Physicians/American Association for Respiratory Care/American College of Critical Care Medicine Task Force on ventilator weaning.


METHODS
Two authors extracted data from all studies included in the Task Force's meta-analysis. DATA SYNTHESIS AND OVERVIEW: The major obstacle to reliable internal validity and, thus, reliable external validity (generalizability) in biological research is systematic error, not random error. If systematic errors are present, averaging (as with a meta-analysis) does not decrease them--instead, it reinforces them, producing artifact. The Task Force's meta-analysis commits several examples of the three main types of systematic error: selection bias (test-referral bias, spectrum bias), misclassification bias (categorizing reintubation as weaning failure, etc.), and confounding (pressure support treated as unassisted breathing). Several additional interpretative errors are present.


CONCLUSIONS
An increase in study size, as achieved through the pooling of data in a meta-analysis, is mistakenly thought to increase external validity. On the contrary, combining heterogeneous studies poses considerable risk of systematic error, which impairs internal validity and, thus, external validity. The strength of recommendations in clinical practice guidelines is based on a misperception of the relative importance of systematic vs. random error in science.
Tobin
Martin J
MJ
Division of Pulmonary and Critical Care Medicine, Edward Hines Jr. Veterans Affairs Hospital, Hines, IL, USA. mtobin2@lumc.edu
Jubran
Amal
A
D015986:Confounding Factors, Epidemiologic; D019317:Evidence-Based Medicine / Q000379:methods; D006801:Humans; D015201:Meta-Analysis as Topic; D011361:Professional Competence; D015203:Reproducibility of Results; D012196:Review Literature as Topic; D018401:Sample Size; D015983:Selection Bias; D015300:Ventilator Weaning