title: Validity of self-assessment in a quality improvement collaborative in Ecuador.
abstract: OBJECTIVE
  Health care quality improvement (QI) efforts commonly use self-assessment to measure compliance with quality standards. This study investigates the validity of self-assessment of quality indicators.
  
  
  METHODS
  Cross sectional.
  
  
  METHODS
  A maternal and newborn care improvement collaborative intervention conducted in health facilities in Ecuador in 2005.
  
  
  METHODS
  Four external evaluators were trained in abstracting medical records to calculate six indicators reflecting compliance with treatment standards.
  
  
  METHODS
  About 30 medical records per month were examined at 12 participating health facilities for a total of 1875 records. The same records had already been reviewed by QI teams at these facilities (self-assessment).
  
  
  METHODS
  Overall compliance, agreement (using the Kappa statistic), sensitivity and specificity were analyzed. We also examined patterns of disagreement and the effect of facility characteristics on levels of agreement.
  
  
  RESULTS
  External evaluators reported compliance of 69-90%, while self-assessors reported 71-92%, with raw agreement of 71-95% and Kappa statistics ranging from fair to almost perfect agreement. Considering external evaluators as the gold standard, sensitivity of self-assessment ranged from 90 to 99% and specificity from 48 to 86%. Simpler indicators had fewer disagreements. When disagreements occurred between self-assessment and external valuators, the former tended to report more positive findings in five of six indicators, but this tendency was not of a magnitude to change program actions. Team leadership, understanding of the tools and facility size had no overall impact on the level of agreement.
  
  
  CONCLUSIONS
  When compared with external evaluation (gold standard), self-assessment was found to be sufficiently valid for tracking QI team performance. Sensitivity was generally higher than specificity. Simplifying indicators may improve validity.
authors.affiliation: University Research Co, LLC/HCI, Avenida ColonE8-57 y Diego Almagro, Edificio El Cisne, Sto Piso Oficina C-D, Quito, Ecuador. jhermida@urc-chs.com
authors.forename: Jorge
authors.initials: J
authors.lastname: Hermida
authors.forename: Edward I
authors.initials: EI
authors.lastname: Broughton
authors.forename: Lynne
authors.initials: L
authors.lastname: Miller Franco
mesh_terms: D003151:Community Health Centers / Q000592:standards; D003299:Cooperative Behavior; D003430:Cross-Sectional Studies; D004484:Ecuador; D005260:Female; D019983:Guideline Adherence; D006801:Humans; D007231:Infant, Newborn; D008427:Maternal Health Services; D008485:Medical Audit; D017283:Neonatal Nursing / Q000592:standards; D009775:Obstetrics and Gynecology Department, Hospital / Q000592:standards; D011247:Pregnancy; D011785:Quality Assurance, Health Care / Q000592:standards*; D019984:Quality Indicators, Health Care; D015203:Reproducibility of Results