Exploring interaction effects in small samples increases rates of false-positive and false-negative findings: results from a systematic review and simulation study.
OBJECTIVE
To give a comprehensive comparison of the performance of commonly applied interaction tests.


METHODS
A literature review and simulation study was performed evaluating interaction tests on the odds ratio (OR) or the risk difference (RD) scales: Cochran Q (Q), Breslow-Day (BD), Tarone, unconditional score, likelihood ratio (LR), Wald, and relative excess risk due to interaction (RERI)-based tests.


RESULTS
Review results agreed with results from our simulation study, which showed that on the OR scale, in small sample sizes (eg, number of subjects ≤ 250) the type 1 error rates of the LR test was 0.10; the BD and Tarone tests showed results around 0.05. On the RD scale, the LR and RERI tests had error rates around 0.05. On both scales, tests did not differ regarding power. When exposure prevented the outcome RERI-based tests were relatively underpowered (eg, N = 100; RERI power = 5% vs. Wald power = 18%). With increasing sample size, difference decreased.


CONCLUSIONS
In small samples, interaction tests differed. On the OR scale, the Tarone and BD tests are recommended. On the RD scale, the LR and RERI-based tests performed best. However, RERI-based tests are underpowered compared with other tests, when exposure prevents the outcome, and sample size is limited.
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands; Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB Utrecht, The Netherlands; Department of Farm Animal Health, Faculty of Veterinary Medicine, Utrecht University, Yalelaan 107, 3584 CL Utrecht, The Netherlands. Electronic address: a.f.schmidt@umcutrecht.nl.
Amand F
AF
Schmidt
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands; Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB Utrecht, The Netherlands.
Rolf H H
RH
Groenwold
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands; RIVM, National Institute for Public Health and the Environment Centre for Infectious Disease Control Netherlands (CIb) Epidemiology and Surveillance Unit (EPI), P.O. Box 1, 3720 BA Bilthoven, The Netherlands.
Mirjam J
MJ
Knol
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands.
Arno W
AW
Hoes
Department of Farm Animal Health, Faculty of Veterinary Medicine, Utrecht University, Yalelaan 107, 3584 CL Utrecht, The Netherlands.
Mirjam
M
Nielen
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands.
Kit C B
KC
Roes
Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB Utrecht, The Netherlands.
Anthonius
A
de Boer
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands; Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB Utrecht, The Netherlands.
Olaf H
OH
Klungel
D003198:Computer Simulation; D003951:Diagnostic Errors; D004812:Epidemiologic Methods; D005188:False Negative Reactions; D005189:False Positive Reactions; D006801:Humans; D016017:Odds Ratio; D012306:Risk; D018401:Sample Size; D013223:Statistics as Topic
Effect modification; Epidemiologic methods; Interaction; Odds ratio; Relative excess risk due to interaction; Review; Risk ratio; Simulation; Statistics; Subgroups