title: Double sampling with multiple imputation to answer large sample meta-research questions: introduction and illustration by evaluating adherence to two simple CONSORT guidelines.
abstract: BACKGROUND
  Meta-research can involve manual retrieval and evaluation of research, which is resource intensive. Creation of high throughput methods (e.g., search heuristics, crowdsourcing) has improved feasibility of large meta-research questions, but possibly at the cost of accuracy.
  
  
  OBJECTIVE
  To evaluate the use of double sampling combined with multiple imputation (DS + MI) to address meta-research questions, using as an example adherence of PubMed entries to two simple consolidated standards of reporting trials guidelines for titles and abstracts.
  
  
  METHODS
  For the DS large sample, we retrieved all PubMed entries satisfying the filters: RCT, human, abstract available, and English language (n = 322, 107). For the DS subsample, we randomly sampled 500 entries from the large sample. The large sample was evaluated with a lower rigor, higher throughput (RLOTHI) method using search heuristics, while the subsample was evaluated using a higher rigor, lower throughput (RHITLO) human rating method. Multiple imputation of the missing-completely at-random RHITLO data for the large sample was informed by: RHITLO data from the subsample; RLOTHI data from the large sample; whether a study was an RCT; and country and year of publication.
  
  
  RESULTS
  The RHITLO and RLOTHI methods in the subsample largely agreed (phi coefficients: title = 1.00, abstract = 0.92). Compliance with abstract and title criteria has increased over time, with non-US countries improving more rapidly. DS + MI logistic regression estimates were more precise than subsample estimates (e.g., 95% CI for change in title and abstract compliance by year: subsample RHITLO 1.050-1.174 vs. DS + MI 1.082-1.151). As evidence of improved accuracy, DS + MI coefficient estimates were closer to RHITLO than the large sample RLOTHI.
  
  
  CONCLUSIONS
  Our results support our hypothesis that DS + MI would result in improved precision and accuracy. This method is flexible and may provide a practical way to examine large corpora of literature.
authors.affiliation: Office of Energetics and Nutrition Obesity Research Center, School of Public Health, University of Alabama at Birmingham , Birmingham, AL , USA.
authors.forename: Patrice L
authors.initials: PL
authors.lastname: Capers
authors.affiliation: Office of Energetics and Nutrition Obesity Research Center, School of Public Health, University of Alabama at Birmingham , Birmingham, AL , USA.
authors.forename: Andrew W
authors.initials: AW
authors.lastname: Brown
authors.affiliation: Office of Energetics and Nutrition Obesity Research Center, School of Public Health, University of Alabama at Birmingham , Birmingham, AL , USA ; Section on Statistical Genetics, University of Alabama at Birmingham , Birmingham, AL , USA.
authors.forename: John A
authors.initials: JA
authors.lastname: Dawson
authors.affiliation: Office of Energetics and Nutrition Obesity Research Center, School of Public Health, University of Alabama at Birmingham , Birmingham, AL , USA ; Section on Statistical Genetics, University of Alabama at Birmingham , Birmingham, AL , USA ; Department of Nutrition Sciences, University of Alabama at Birmingham , Birmingham, AL , USA ; Department of Biostatistics, University of Alabama at Birmingham , Birmingham, AL , USA.
authors.forename: David B
authors.initials: DB
authors.lastname: Allison
keywords: CONSORT; adherence; double sampling; meta-research; modeling; multiple imputation