title: A case report of evaluating a large-scale health systems improvement project in an uncontrolled setting: a quality improvement initiative in KwaZulu-Natal, South Africa.
abstract: OBJECTIVE
  New approaches are needed to evaluate quality improvement (QI) within large-scale public health efforts. This case report details challenges to large-scale QI evaluation, and proposes solutions relying on adaptive study design.
  
  
  METHODS
  We used two sequential evaluative methods to study a QI effort to improve delivery of HIV preventive care in public health facilities in three districts in KwaZulu-Natal, South Africa, over a 3-year period. We initially used a cluster randomised controlled trial (RCT) design.
  
  
  RESULTS
  During the RCT study period, tensions arose between intervention implementation and evaluation design due to loss of integrity of the randomisation unit over time, pressure to implement changes across the randomisation unit boundaries, and use of administrative rather than functional structures for the randomisation. In response to this loss of design integrity, we switched to a more flexible intervention design and a mixed-methods quasiexperimental evaluation relying on both a qualitative analysis and an interrupted time series quantitative analysis.
  
  
  CONCLUSIONS
  Cluster RCT designs may not be optimal for evaluating complex interventions to improve implementation in uncontrolled 'real world' settings. More flexible, context-sensitive evaluation designs offer a better balance of the need to adjust the intervention during the evaluation to meet implementation challenges while providing the data required to evaluate effectiveness. Our case study involved HIV care in a resource-limited setting, but these issues likely apply to complex improvement interventions in other settings.
authors.affiliation: Institute for Healthcare Improvement, , Cambridge, Massachusetts USA.
authors.forename: Kedar S
authors.initials: KS
authors.lastname: Mate
authors.forename: Wilbroda Hlolisile
authors.initials: WH
authors.lastname: Ngidi
authors.forename: Jennifer
authors.initials: J
authors.lastname: Reddy
authors.forename: Wendy
authors.initials: W
authors.lastname: Mphatswe
authors.forename: Nigel
authors.initials: N
authors.lastname: Rollins
authors.forename: Pierre
authors.initials: P
authors.lastname: Barker
mesh_terms: D005260:Female; D015658:HIV Infections / Q000517:prevention & control* / Q000635:transmission; D006801:Humans; D007231:Infant, Newborn; D018445:Infectious Disease Transmission, Vertical / Q000517:prevention & control*; D009042:Motivation; D011247:Pregnancy; D015397:Program Evaluation; D011634:Public Health; D058996:Quality Improvement; D012107:Research Design; D013019:South Africa