title: Benchmarking physician performance: reliability of individual and composite measures.
abstract: OBJECTIVE
  To examine the reliability of quality measures to assess physician performance, which are increasingly used as the basis for quality improvement efforts, contracting decisions, and financial incentives, despite concerns about the methodological challenges.
  
  
  METHODS
  Evaluation of health plan administrative claims and enrollment data.
  
  
  METHODS
  The study used administrative data from 9 health plans representing more than 11 million patients. The number of quality events (patients eligible for a quality measure), mean performance, and reliability estimates were calculated for 27 quality measures. Composite scores for preventive, chronic, acute, and overall care were calculated as the weighted mean of the standardized scores. Reliability was estimated by calculating the physician-to-physician variance divided by the sum of the physician-to-physician variance plus the measurement variance, and 0.70 was considered adequate.
  
  
  RESULTS
  Ten quality measures had reliability estimates above 0.70 at a minimum of 50 quality events. For other quality measures, reliability was low even when physicians had 50 quality events. The largest proportion of physicians who could be reliably evaluated on a single quality measure was 8% for colorectal cancer screening and 2% for nephropathy screening among patients with diabetes mellitus. More physicians could be reliably evaluated using composite scores (<17% for preventive care, >7% for chronic care, and 15%-20% for an overall composite).
  
  
  CONCLUSIONS
  In typical health plan administrative data, most physicians do not have adequate numbers of quality events to support reliable quality measurement. The reliability of quality measures should be taken into account when quality information is used for public reporting and accountability. Efforts to improve data available for physician profiling are also needed.
authors.affiliation: National Committee for Quality Assurance, Washington, DC 20005, USA. scholle@ncqa.org
authors.forename: Sarah Hudson
authors.initials: SH
authors.lastname: Scholle
authors.forename: Joachim
authors.initials: J
authors.lastname: Roski
authors.forename: John L
authors.initials: JL
authors.lastname: Adams
authors.forename: Daniel L
authors.initials: DL
authors.lastname: Dunn
authors.forename: Eve A
authors.initials: EA
authors.lastname: Kerr
authors.forename: Donna Pillittere
authors.initials: DP
authors.lastname: Dugan
authors.forename: Roxanne E
authors.initials: RE
authors.lastname: Jensen
mesh_terms: D000465:Algorithms; D019985:Benchmarking / Q000379:methods*; D017723:Drug Utilization Review; D019538:Health Care Surveys; D006801:Humans; D033181:Information Dissemination; D008329:Managed Care Programs / Q000592:standards*; D008485:Medical Audit / Q000379:methods*; D010820:Physicians / Q000145:classification / Q000592:standards*; D011322:Primary Prevention / Q000592:standards; D019984:Quality Indicators, Health Care / Q000145:classification*; D015203:Reproducibility of Results; D012941:Social Responsibility; D017749:Total Quality Management / Q000379:methods*; D014481:United States