title: Funnel plots for comparing provider performance based on patient-reported outcome measures.
abstract: BACKGROUND
  Patient-reported outcome measures (PROMs) often produce skewed distributions of individual scores after a healthcare intervention. For health performance indicators derived from skewed distributions, funnel plots designed with symmetric control limits may increase the risk of false alarms about poor performance.
  
  
  OBJECTIVE
  To investigate the accuracy of funnel plots with symmetric control limits when comparing provider performance based on PROMs.
  
  
  METHODS
  The authors used a database containing condition-specific PROMs for 17,453 hip replacements and 7656 varicose vein procedures performed by providers in the English NHS. The mean postoperative PROM score, adjusted for patient characteristics, was used as the measure of performance. To compare performance, symmetric 99.8% control limits were calculated on funnel plots, 3 SDs away from the overall mean on either side. These were compared to control limits derived directly from percentiles of simulated (bootstrap) distributions of mean scores.
  
  
  RESULTS
  The simulated control limits on funnel plots for both procedures were asymmetric. The empirical probability of falling outside the symmetric 99.8% 'poor performance' control limit was inflated from the stipulated rate of 0.1% to 0.2-0.3% for provider sample sizes of up to 150 procedures. The authors observed that, out of 237 providers of hip replacement, eight had adjusted mean scores that exceeded the symmetric 'poor performance' limit compared with only five that exceeded the corresponding simulated limit. In other words, three (1.3%) were differently classified. For varicose vein surgery, five out of 160 providers exceeded the symmetric limit and four exceeded the simulated limit, that is, 1 (0.6%) was differently classified.
  
  
  CONCLUSIONS
  When designing funnel plots for comparisons of provider performance based on highly skewed data, the use of simulated control limits should be considered.
authors.affiliation: Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, London WC1H 9SH, UK. jenny.neuburger@lshtm.ac.uk
authors.forename: Jenny
authors.initials: J
authors.lastname: Neuburger
authors.forename: David A
authors.initials: DA
authors.lastname: Cromwell
authors.forename: Andrew
authors.initials: A
authors.lastname: Hutchings
authors.forename: Nick
authors.initials: N
authors.lastname: Black
authors.forename: Jan H
authors.initials: JH
authors.lastname: van der Meulen
mesh_terms: D019644:Arthroplasty, Replacement, Hip; D002983:Clinical Competence / Q000706:statistics & numerical data; D019992:Databases as Topic; D004739:England / Q000453:epidemiology; D006801:Humans; D017063:Outcome Assessment, Health Care / Q000379:methods*; D017060:Patient Satisfaction; D011183:Postoperative Complications / Q000453:epidemiology; D019984:Quality Indicators, Health Care / Q000706:statistics & numerical data; D011787:Quality of Health Care / Q000592:standards*; D020379:Risk Adjustment; D013222:State Medicine; D011795:Surveys and Questionnaires; D014648:Varicose Veins / Q000601:surgery