title: Simulation-based evaluation of the performance of the F test in a linear multilevel model setting with sparseness at the level of the primary unit.
abstract: In a linear multilevel model, significance of all fixed effects can be determined using F tests under maximum likelihood (ML) or restricted maximum likelihood (REML). In this paper, we demonstrate that in the presence of primary unit sparseness, the performance of the F test under both REML and ML is rather poor. Using simulations based on the structure of a data example on ceftriaxone consumption in hospitalized children, we studied variability, type I error rate and power in scenarios with a varying number of secondary units within the primary units. In general, the variability in the estimates for the effect of the primary unit decreased as the number of secondary units increased. In the presence of singletons (i.e., only one secondary unit within a primary unit), REML consistently outperformed ML, although even under REML the performance of the F test was found inadequate. When modeling the primary unit as a random effect, the power was lower while the type I error rate was unstable. The options of dropping, regrouping, or splitting the singletons could solve either the problem of a high type I error rate or a low power, while worsening the other. The permutation test appeared to be a valid alternative as it outperformed the F test, especially under REML. We conclude that in the presence of singletons, one should be careful in using the F test to determine the significance of the fixed effects, and propose the permutation test (under REML) as an alternative.
authors.affiliation: Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-BIOSTAT), Hasselt University, Martelarenlaan 42, Hasselt, Belgium. robin.bruyndonckx@uhasselt.be.
authors.forename: Robin
authors.initials: R
authors.lastname: Bruyndonckx
authors.affiliation: Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-BIOSTAT), Hasselt University, Martelarenlaan 42, Hasselt, Belgium.
authors.forename: Marc
authors.initials: M
authors.lastname: Aerts
authors.affiliation: Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-BIOSTAT), Hasselt University, Martelarenlaan 42, Hasselt, Belgium.|Centre for Health Economic Research and Modelling of Infectious Diseases (CHERMID), Vaccine & Infectious Disease Institute (VAXINFECTIO), University of Antwerp, Universiteitsplein 1, Wilrijk, Belgium.
authors.forename: Niel
authors.initials: N
authors.lastname: Hens
mesh_terms: D002443:Ceftriaxone / Q000600:supply & distribution; D002648:Child; D003198:Computer Simulation; D006801:Humans; D016013:Likelihood Functions; D008962:Models, Theoretical
chemical_list: D002443:Ceftriaxone
keywords: F test; Maximum likelihood; Permutation test; Primary unit sparseness; Restricted maximum likelihood