In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias.
OBJECTIVE
Publication bias and related biases can lead to overly optimistic conclusions in systematic reviews. The funnel plot, which is frequently used to detect such biases, has not yet been subjected to empirical evaluation as a visual tool. We sought to determine whether researchers can correctly identify publication bias from visual inspection of funnel plots in typical-size systematic reviews.


METHODS
A questionnaire with funnel plots containing 10 studies each (the median number in medical meta-analyses) was completed by 41 medical researchers, including clinical research fellows in a meta-analysis class, faculty in clinical care research, and experienced systematic reviewers.


RESULTS
On average, participants correctly identified 52.5% (95% CI 50.6-54.4%) of the plots as being affected or unaffected by publication bias. The weighted mean percent correct, which adjusted for the fact that asymmetric plots are more likely to occur in the presence of publication bias, was also low (48.3 to 62.8%, depending on the presence or absence of publication bias and heterogeneous study effects).


CONCLUSIONS
Researchers who assess for publication bias using the funnel plot may be misled by its shape. Authors and readers of systematic reviews need to be aware of the limitations of the funnel plot.
Terrin
Norma
N
Institute for Clinical Research and Health Policy Studies, Tufts-New England Medical Center, 750 Washington Street, Box 63, Boston, MA 02111, USA. nterrin@tufts-nemc.org
Schmid
Christopher H
CH
Lau
Joseph
J
D006801:Humans; D015201:Meta-Analysis as Topic; D016017:Odds Ratio; D017594:Publication Bias; D013223:Statistics as Topic / Q000379:methods*; D011795:Surveys and Questionnaires