title: Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda.
abstract: Data collection techniques that routinely provide health system information at the local level are in demand and needed. LQAS is intended for use by local health teams to collect data at the district and sub-district levels. Our question is whether local health staff produce biased results as they are responsible for implementing the programs they also assess.
  
  
  
  This test-retest study replicates on a larger scale an earlier LQAS reliability assessment in Uganda. We conducted in two districts an LQAS survey using 15 local health staff as data collectors. A week later, the data collectors swapped districts, where they acted as disinterested non-local data collectors, repeating the LQAS survey with the same respondents. We analysed the resulting two data sets for agreement using Cohens' Kappa.
  
  
  
  The average Kappa score for the knowledge indicators was k = 0.43 (SD = 0.16) and for practice indicators k = 0.63 (SD = 0.17). These scores show moderate agreement for knowledge indicators and substantial agreement for practice indicators. Analyses confirm that respondents were more knowledgeable on retest; no evidence of bias was found for practice indicators.
  
  
  
  The findings of this study are remarkably similar to those produced in the first reliability study. There is no evidence that using local healthcare staff to collect LQAS data biases data collection in an LQAS study. The bias observed in the knowledge indicators was most likely due to a 'practice effect', whereby respondents increased their knowledge as a result of completing the first survey; no corresponding effect was seen in the practice indicators.
authors.affiliation: Liverpool School of Tropical Medicine, Pembroke Place, Liverpool, L3 5QA, United Kingdom. cbeckworth74@gmail.com.
authors.forename: Colin A
authors.initials: CA
authors.lastname: Beckworth
authors.affiliation: Uganda Martyrs University, Nkozi, Uganda.
authors.forename: Robert
authors.initials: R
authors.lastname: Anguyo
authors.affiliation: Uganda Christian University, Mukono, Uganda.
authors.forename: Francis Cranmer
authors.initials: FC
authors.lastname: Kyakulaga
authors.affiliation: Management Sciences for Health, USAID STAR-E project, Mukono, Uganda.
authors.forename: Stephen K
authors.initials: SK
authors.lastname: Lwanga
authors.affiliation: Liverpool School of Tropical Medicine, Pembroke Place, Liverpool, L3 5QA, United Kingdom.
authors.forename: Joseph J
authors.initials: JJ
authors.lastname: Valadez
mesh_terms: D015982:Bias; D003695:Delivery of Health Care / Q000592:standards; D006282:Health Personnel / Q000592:standards; D006801:Humans; D055810:Lot Quality Assurance Sampling; D015588:Observer Variation; D011785:Quality Assurance, Health Care / Q000379:methods*; D019984:Quality Indicators, Health Care / Q000592:standards; D015203:Reproducibility of Results; D012494:Sampling Studies; D011795:Surveys and Questionnaires; D014454:Uganda
keywords: Bias; Cohen’s kappa; LQAS; Lot Quality Assurance Sampling; Test retest