title: The Influence of Judgment Calls on Meta-Analytic Findings.
abstract: Previous research has suggested that judgment calls (i.e., methodological choices made in the process of conducting a meta-analysis) have a strong influence on meta-analytic findings and question their robustness. However, prior research applies case study comparison or reanalysis of a few meta-analyses with a focus on a few selected judgment calls. These studies neglect the fact that different judgment calls are related to each other and simultaneously influence the outcomes of a meta-analysis, and that meta-analytic findings can vary due to non-judgment call differences between meta-analyses (e.g., variations of effects over time). The current study analyzes the influence of 13 judgment calls in 176 meta-analyses in marketing research by applying a multivariate, multilevel meta-meta-analysis. The analysis considers simultaneous influences from different judgment calls on meta-analytic effect sizes and controls for alternative explanations based on non-judgment call differences between meta-analyses. The findings suggest that judgment calls have only a minor influence on meta-analytic findings, whereas non-judgment call differences between meta-analyses are more likely to explain differences in meta-analytic findings. The findings support the robustness of meta-analytic results and conclusions.
authors.affiliation: a European University Viadrina.
authors.forename: Farid
authors.initials: F
authors.lastname: Tarrahi
authors.affiliation: a European University Viadrina.
authors.forename: Martin
authors.initials: M
authors.lastname: Eisend
mesh_terms: D003627:Data Interpretation, Statistical; D006801:Humans; D007600:Judgment; D015201:Meta-Analysis as Topic; D055361:Multilevel Analysis / Q000379:methods*; D015999:Multivariate Analysis
keywords: Empirical generalization; judgment calls; meta-analysis