title: Comparisons of health plan quality: does the sampling strategy affect the results?
abstract: BACKGROUND
  Health plan evaluations typically focus on beneficiaries continuously enrolled throughout a calendar year.
  
  
  OBJECTIVE
  To examine whether the sampling strategy used in evaluating health plans affects their assessment.
  
  
  METHODS
  We completed a cross-sectional analysis of California Medicaid managed care plans aggregating individual level data by health plan.
  
  
  METHODS
  We calculated the correlation between a health plan's annual age-sex adjusted hospitalization rate for ambulatory care sensitive (ACS) conditions sampling among beneficiaries continuously enrolled throughout the year with all health plan beneficiaries enrolled for a minimum of 1 month. For health plan beneficiaries with a minimum of 1 month of enrollment we calculated hospitalization rates in 2 ways: (1) counting only those ACS hospitalizations while enrolled in the plan and (2) counting any ACS hospitalization throughout the year regardless of whether it was during the period of plan enrollment.
  
  
  METHODS
  California Medicaid beneficiaries 18-64 years of age in 2001.
  
  
  RESULTS
  Forty-four Medicaid plans representing approximately 750,000 beneficiaries were included. On average, 50% (range, 26-69% across plans) of the beneficiaries were enrolled continuously during the year. Plan rankings based on the ACS hospitalization rates for each of the 3 sampling strategies were variably correlated [Spearman correlations 0.26 (P = 0.086), 0.33 (P = 0.031), and 0.71 (P < 0.0001) for pairwise comparisons]. The agreement among the sampling strategies in labeling a health plan as an outlier was not statistically different than random chance (kappa = 0.069, P = 0.21).
  
  
  CONCLUSIONS
  Judgments regarding health plan performance are affected by limiting the evaluation sample to only those beneficiaries continuously enrolled in a health plan throughout the year. Policymakers should consider the goal of health care measurement when selecting a sampling strategy and explicitly acknowledge the bias that might be introduced by a particular approach.
authors.affiliation: Department of Medicine, University of California, San Francisco, CA 94143, USA. GoldmanL@medicine.ucsf.edu
authors.forename: L Elizabeth
authors.initials: LE
authors.lastname: Goldman
authors.forename: Arpita
authors.initials: A
authors.lastname: Chattopadhyay
authors.forename: Andrew B
authors.initials: AB
authors.lastname: Bindman
mesh_terms: D000293:Adolescent; D000328:Adult; D000553:Ambulatory Care; D002140:California; D003430:Cross-Sectional Studies; D005260:Female; D006760:Hospitalization; D006801:Humans; D008297:Male; D008329:Managed Care Programs / Q000592:standards*; D008484:Medicaid; D008875:Middle Aged; D011787:Quality of Health Care; D020379:Risk Adjustment; D014481:United States