Empirically supported treatments or type I errors? Problems with the analysis of data from group-administered treatments.
When treatments are administered in groups, clients interact in ways that lead to violations of a key assumption of most statistical analyses-the assumption of independence of observations. The resulting dependencies, when not properly accounted for, can increase Type I errors dramatically. Of the 33 studies of group-administered treatment on the empirically supported treatments list, none appropriately analyzed their data. The current authors provide corrections that can be applied to improper analyses. After the corrections, only 12.4% to 68.2% of tests that were originally reported as significant remained significant, depending on what assumptions were made about how large the dependencies among observations really are. Of the 33 studies, 6-19 studies no longer had any significant results after correction. The authors end by providing recommendations for researchers planning group-administered treatment research.
Baldwin
Scott A
SA
Department of Psychology, University of Memphis, Memphis, TN, USA. s.baldwin@mac.com
Murray
David M
DM
Shadish
William R
WR
D000704:Analysis of Variance; D001521:Behavior Therapy / Q000592:standards* / Q000706:statistics & numerical data; D015982:Bias; D003627:Data Interpretation, Statistical; D036262:Empirical Research; D019317:Evidence-Based Medicine; D006801:Humans; D011615:Psychotherapy, Group / Q000592:standards* / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D012107:Research Design; D016896:Treatment Outcome