Benchmarking physician performance: reliability of individual and composite measures.
OBJECTIVE
To examine the reliability of quality measures to assess physician performance, which are increasingly used as the basis for quality improvement efforts, contracting decisions, and financial incentives, despite concerns about the methodological challenges.


METHODS
Evaluation of health plan administrative claims and enrollment data.


METHODS
The study used administrative data from 9 health plans representing more than 11 million patients. The number of quality events (patients eligible for a quality measure), mean performance, and reliability estimates were calculated for 27 quality measures. Composite scores for preventive, chronic, acute, and overall care were calculated as the weighted mean of the standardized scores. Reliability was estimated by calculating the physician-to-physician variance divided by the sum of the physician-to-physician variance plus the measurement variance, and 0.70 was considered adequate.


RESULTS
Ten quality measures had reliability estimates above 0.70 at a minimum of 50 quality events. For other quality measures, reliability was low even when physicians had 50 quality events. The largest proportion of physicians who could be reliably evaluated on a single quality measure was 8% for colorectal cancer screening and 2% for nephropathy screening among patients with diabetes mellitus. More physicians could be reliably evaluated using composite scores (<17% for preventive care, >7% for chronic care, and 15%-20% for an overall composite).


CONCLUSIONS
In typical health plan administrative data, most physicians do not have adequate numbers of quality events to support reliable quality measurement. The reliability of quality measures should be taken into account when quality information is used for public reporting and accountability. Efforts to improve data available for physician profiling are also needed.
Scholle
Sarah Hudson
SH
National Committee for Quality Assurance, Washington, DC 20005, USA. scholle@ncqa.org
Roski
Joachim
J
Adams
John L
JL
Dunn
Daniel L
DL
Kerr
Eve A
EA
Dugan
Donna Pillittere
DP
Jensen
Roxanne E
RE
D000465:Algorithms; D019985:Benchmarking / Q000379:methods*; D017723:Drug Utilization Review; D019538:Health Care Surveys; D006801:Humans; D033181:Information Dissemination; D008329:Managed Care Programs / Q000592:standards*; D008485:Medical Audit / Q000379:methods*; D010820:Physicians / Q000145:classification / Q000592:standards*; D011322:Primary Prevention / Q000592:standards; D019984:Quality Indicators, Health Care / Q000145:classification*; D015203:Reproducibility of Results; D012941:Social Responsibility; D017749:Total Quality Management / Q000379:methods*; D014481:United States