title: Profiling individual physicians using administrative data from a single insurer: variance components, reliability, and implications for performance improvement efforts.
abstract: BACKGROUND
  Individual physicians are increasingly being subjected to comparative performance assessments. When single-insurer data are used to profile individual physicians' performance, reliable measurements are uncertain because of small sample sizes.
  
  
  METHODS
  Administrative data (2006-2008) from a Dutch insurer are used to examine variation in general practitioners' (GPs) performance on expenses (5 measures), utilization of hospital care (2 measures), and clinical quality for diabetes and chronic obstructive pulmonary disease (6 measures). Unadjusted and adjusted multilevel models are used to separate total variance in between-GP and within-GP components. The components are used to calculate intraclass correlation coefficients (ICCs), reliability, and sample size requirements at common reliability thresholds.
  
  
  RESULTS
  Average ICCs varied between 0.07% (hospital admissions) and 8.34% (physiotherapy for chronic obstructive pulmonary disease patients). Risk-adjustment often greatly changed the relative size of variance components and often led to lower ICCs. In addition, ICCs and thus reliability generally decreased over time. Eight measures had reliabilities > 0.70, and 3 of these (all GP-related expenses) > 0.90. Measures related to utilization of hospital care had reliabilities < 0.60 or even 0.50. For 5 measures, the vast majority of GPs had sufficient patients to reach 0.70 reliability. At a reliability of 0.90, however, there were no measures for which all GPs met the sample size requirements.
  
  
  CONCLUSIONS
  Reliable measurement of individual physicians' performance using single-purchaser data is challenging. For most measures reliability was insufficient to allow for high-stakes applications or even any application of profiling. Future research should continue to explore methods for enhancing the reliability of individual physicians' profiles.
authors.affiliation: Institute of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands. eijkenaar@bmg.eur.nl
authors.forename: Frank
authors.initials: F
authors.lastname: Eijkenaar
authors.forename: René C J A
authors.initials: RC
authors.lastname: van Vliet
mesh_terms: D000328:Adult; D000367:Age Factors; D002983:Clinical Competence; D003625:Data Collection / Q000379:methods*; D003920:Diabetes Mellitus / Q000628:therapy; D004650:Employee Performance Appraisal / Q000379:methods*; D005260:Female; D005102:Health Expenditures / Q000706:statistics & numerical data; D006761:Hospitals / Q000706:statistics & numerical data; D006801:Humans; D007343:Insurance Carriers / Q000706:statistics & numerical data*; D008297:Male; D008875:Middle Aged; D009426:Netherlands; D010820:Physicians / Q000706:statistics & numerical data*; D010818:Practice Patterns, Physicians' / Q000706:statistics & numerical data*; D029424:Pulmonary Disease, Chronic Obstructive / Q000628:therapy; D011787:Quality of Health Care / Q000458:organization & administration / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D020379:Risk Adjustment; D012737:Sex Factors; D012959:Socioeconomic Factors