No short-cut in assessing trial quality: a case study.
BACKGROUND
Assessing the quality of included trials is a central part of a systematic review. Many check-list type of instruments for doing this exist. Using a trial of antibiotic treatment for acute otitis media, Burke et al., BMJ, 1991, as the case study, this paper illustrates some limitations of the check-list approach to trial quality assessment.


RESULTS
The general verdict from the check list type evaluations in nine relevant systematic reviews was that Burke et al. (1991) is a good quality trial. All relevant meta-analyses extensively used its data to formulate therapeutic evidence. My comprehensive evaluation, on the other hand, brought to the surface a series of serious problems in the design, conduct, analysis and report of this trial that were missed by the earlier evaluations.


CONCLUSIONS
A check-list or instrument based approach, if used as a short-cut, may at times rate deeply flawed trials as good quality trials. Check lists are crucial but they need to be augmented with an in-depth review, and where possible, a scrutiny of the protocol, trial records, and original data. The extent and severity of the problems I uncovered for this particular trial warrant an independent audit before it is included in a systematic review.
Hirji
Karim F
KF
Department of Epidemiology and Biostatistics, Muhimbili University of Health and Allied Sciences, P O Box 65015, Dar es Salaam, Tanzania. kfhirji@aol.com
D022126:Access to Information; D000208:Acute Disease; D000293:Adolescent; D000658:Amoxicillin / Q000627:therapeutic use*; D000900:Anti-Bacterial Agents / Q000627:therapeutic use*; D035843:Biomedical Research / Q000592:standards*; D002648:Child; D002675:Child, Preschool; D004311:Double-Blind Method; D019317:Evidence-Based Medicine; D006801:Humans; D020407:Internet; D015201:Meta-Analysis as Topic; D010033:Otitis Media / Q000188:drug therapy*; D017712:Peer Review, Research; D010506:Periodicals as Topic; D017594:Publication Bias; D011786:Quality Control; D016032:Randomized Controlled Trials as Topic / Q000592:standards*; D015203:Reproducibility of Results; D012107:Research Design / Q000592:standards*; D015871:Scientific Misconduct; D016896:Treatment Outcome
D000900:Anti-Bacterial Agents; D000658:Amoxicillin