Increased risk of type I errors in cluster randomised trials with small or medium numbers of clusters: a review, reanalysis, and simulation study.
Cluster randomised trials (CRTs) are commonly analysed using mixed-effects models or generalised estimating equations (GEEs). However, these analyses do not always perform well with the small number of clusters typical of most CRTs. They can lead to increased risk of a type I error (finding a statistically significant treatment effect when it does not exist) if appropriate corrections are not used.



We conducted a small simulation study to evaluate the impact of using small-sample corrections for mixed-effects models or GEEs in CRTs with a small number of clusters. We then reanalysed data from TRIGGER, a CRT with six clusters, to determine the effect of using an inappropriate analysis method in practice. Finally, we reviewed 100 CRTs previously identified by a search on PubMed in order to assess whether trials were using appropriate methods of analysis. Trials were classified as at risk of an increased type I error rate if they did not report using an analysis method which accounted for clustering, or if they had fewer than 40 clusters and performed an individual-level analysis without reporting the use of an appropriate small-sample correction.



Our simulation study found that using mixed-effects models or GEEs without an appropriate correction led to inflated type I error rates, even for as many as 70 clusters. Conversely, using small-sample corrections provided correct type I error rates across all scenarios. Reanalysis of the TRIGGER trial found that inappropriate methods of analysis gave much smaller P values (P ≤ 0.01) than appropriate methods (P = 0.04-0.15). In our review, of the 99 trials that reported the number of clusters, 64 (65 %) were at risk of an increased type I error rate; 14 trials did not report using an analysis method which accounted for clustering, and 50 trials with fewer than 40 clusters performed an individual-level analysis without reporting the use of an appropriate correction.



CRTs with a small or medium number of clusters are at risk of an inflated type I error rate unless appropriate analysis methods are used. Investigators should consider using small-sample corrections with mixed-effects models or GEEs to ensure valid results.
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK. b.kahan@qmul.ac.uk.
Brennan C
BC
Kahan
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK.
Gordon
G
Forbes
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK.
Yunus
Y
Ali
Department of Medicine, Western University and London Health Sciences Network, London, ON, Canada.|Division of Epidemiology and Biostatistics, Western University, London, ON, Canada.
Vipul
V
Jairath
Division of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK.
Stephen
S
Bremner
Division of Epidemiology, Department of Biostatistics and Medicine, University of Pennsylvania, Philadelphia, PA, USA.
Michael O
MO
Harhay
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK.
Richard
R
Hooper
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK.
Neil
N
Wright
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK.
Sandra M
SM
Eldridge
Pragmatic Clinical Trials Unit, Queen Mary University of London, 58 Turner St, E1 2AB, London, UK.|INSERM CIC 1415, CHRU de Tours, Tours, France.|London School of Hygiene and Tropical Medicine, London, UK.
Clémence
C
Leyrat
D016000:Cluster Analysis; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D006801:Humans; D016032:Randomized Controlled Trials as Topic; D012306:Risk
Cluster randomised trials; Degree-of-freedom corrections; Generalised estimating equations; Mixed-effects models; Small-sample corrections