title: Methodology to improve data quality from chart review in the managed care setting.
abstract: BACKGROUND
  Because inherent variability may exist in data collected by multiple reviewers or from potential difficulties with data abstraction tools, we developed a standardized method of evaluating interrater reliability (IRR) for clinical studies, HEDIS effectiveness of care measures, and onsite/medical record reviews.
  
  
  OBJECTIVE
  To demonstrate the ability of our standardized methods of data collection and analysis of results to determine the extent of agreement between multiple reviewers; identify areas for improvement in data collection procedures; and improve data reliability.
  
  
  METHODS
  A prospective chart review with concurrent IRR.
  
  
  METHODS
  A subsample of patient records included in the Highmark Blue Cross Blue Shield/Keystone Health Plan West basic medical review for each HEDIS measure was selected for the IRR study. An experienced nurse ("gold standard") conducted a blinded concurrent review of these records. Using the kappa statistic (kappa) we evaluated interobserver agreement between results of the onsite reviewers and the "gold standard" from 1997 through 2000. Revised data collection methods and enhanced reviewer training were incorporated for measures showing areas for rater improvement.
  
  
  RESULTS
  Results across years showed excellent IRR for most measures; however, each year 1 or 2 measures showed areas for rater improvement (1997 Papanicolaou kappa = 0.50; 1998 well-child visits 3 to 6 years kappa = 0.37; 1999 comprehensive diabetes kappa = 0.73; high blood pressure kappa = 0.73). After reevaluating these measures, the results of the kappa showed excellent interrater agreement in subsequent years.
  
  
  CONCLUSIONS
  Standardized methods of data collection and evaluation of IRR results provides health plans increased confidence in data collection, statistical analyses, and in reaching conclusions and deriving relevant recommendations.
authors.affiliation: Department of Biostatistics, University of Pittsburgh, Graduates School of Public Health, Pa 15261, USA. lcs3@pitt.edu
authors.forename: Laura D
authors.initials: LD
authors.lastname: Cassidy
authors.forename: Gary M
authors.initials: GM
authors.lastname: Marsh
authors.forename: Mary Kay
authors.initials: MK
authors.lastname: Holleran
authors.forename: Linda S
authors.initials: LS
authors.lastname: Ruhl
mesh_terms: D054598:Blue Cross Blue Shield Insurance Plans; D003211:Concurrent Review; D003625:Data Collection / Q000592:standards; D006801:Humans; D008329:Managed Care Programs / Q000592:standards*; D008485:Medical Audit / Q000379:methods / Q000592:standards*; D008499:Medical Records / Q000592:standards*; D011446:Prospective Studies; D011786:Quality Control; D019984:Quality Indicators, Health Care; D015203:Reproducibility of Results; D014481:United States