Synthesis of observational studies should consider credibility ceilings.
OBJECTIVE
Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations.


METHODS
We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids).


RESULTS
Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations.


CONCLUSIONS
Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.
Salanti
Georgia
G
The Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina, School of Medicine, Ioannina, Greece.
Ioannidis
John P A
JP
D000465:Algorithms; D015982:Bias; D003627:Data Interpretation, Statistical; D015525:Fatty Acids, Omega-3 / Q000008:administration & dosage; D006202:Hair Dyes / Q000009:adverse effects; D017052:Hospital Mortality; D006784:Hospitals, Teaching / Q000706:statistics & numerical data; D006801:Humans; D016013:Likelihood Functions; D008228:Lymphoma, Non-Hodgkin / Q000453:epidemiology; D015201:Meta-Analysis as Topic; D012107:Research Design / Q000592:standards; D012306:Risk
D015525:Fatty Acids, Omega-3; D006202:Hair Dyes