title: Assessing the accuracy of profiling methods for identifying top providers: performance of mental health care providers.
abstract: Provider profiling as a means to describe and compare the performance of health care professionals has gained momentum in the past decade. As a key component of pay-for-performance programs profiling has been increasingly used to identify top-performing providers. However, rigorous examination of the performance of statistical methods for profiling when used to classify top-performing providers is lacking. The objective of this study was to compare the classification accuracy of three methods for identifying providers exceeding performance thresholds and to analyze data on satisfaction with mental health care providers at Group Health Cooperative using these methods. Questionnaire data on patient satisfaction with mental health care providers at Group Health Cooperative was collected between April 2008 and January 2010. Simulated data were used to compare the classification accuracy of alternative statistical methods. We evaluated sensitivity, specificity, and root mean squared error of alternative statistical methods using simulated data. For Group Health providers, we compared agreement of alternative approaches to classification. We found that when between-provider variability in performance was low, all three methods exhibited poor classification accuracy. When used to evaluate mental health care provider performance, we found substantial uncertainty in the estimates and poor agreement across methods. Based on these findings, we recommend providing uncertainty estimates for provider rankings and caution against the use of any classification method when between-provider variability is low.
authors.affiliation: Department of Biostatistics, University of Washington, Seattle, WA, USA.
authors.forename: Victoria Y
authors.initials: VY
authors.lastname: Ding
authors.forename: Rebecca A
authors.initials: RA
authors.lastname: Hubbard
authors.forename: Carolyn M
authors.initials: CM
authors.lastname: Rutter
authors.forename: Gregory E
authors.initials: GE
authors.lastname: Simon
keywords: Monte Carlo methods; Pay-for-performance; Provider profiling; Random effects model; Variance components
references.citation: JAMA. 2000 Apr 12;283(14):1866-74
references.pmid: 10770149
references.citation: J Educ Behav Stat. 2002 Jan 1;27(3):255-270
references.pmid: 19830272
references.citation: Med Care Res Rev. 2006 Apr;63(2):135-57
references.pmid: 16595409
references.citation: Eur J Health Econ. 2002;3(3):145-8
references.pmid: 15609138
references.citation: Circulation. 2007 Mar 27;115(12):1518-27
references.pmid: 17353447
references.citation: Biometrics. 2011 Mar;67(1):270-9
references.pmid: 20528860
references.citation: J Eval Clin Pract. 2001 Feb;7(1):35-45
references.pmid: 11240838
references.citation: N Engl J Med. 2004 Jan 22;350(4):406-10
references.pmid: 14736934
references.citation: BMC Med Res Methodol. 2008 May 12;8:30
references.pmid: 18474094
references.citation: J Am Coll Surg. 2009 Dec;209(6):687-93
references.pmid: 19959035
references.citation: Circulation. 2008 Apr 15;117(15):1955-63
references.pmid: 18391106
references.citation: Med Decis Making. 2003 Nov-Dec;23(6):526-39
references.pmid: 14672113
references.citation: J Health Serv Res Policy. 2003 Apr;8(2):68-74
references.pmid: 12820667
references.citation: J Health Econ. 2000 May;19(3):291-309
references.pmid: 10977193
references.citation: J Behav Health Serv Res. 2012 Jan;39(1):42-54
references.pmid: 21847711
references.citation: BMC Med Res Methodol. 2007 Jul 03;7:29
references.pmid: 17608921
references.citation: Ann Intern Med. 2006 Aug 15;145(4):265-72
references.pmid: 16908917
references.citation: Psychol Bull. 1982 Mar;91(2):244-59
references.pmid: 7071260
references.citation: JAMA. 2007 Feb 21;297(7):740-4
references.pmid: 17312294
references.citation: Psychiatr Serv. 2010 Jul;61(7):684-9
references.pmid: 20592003
references.citation: Am Stat. 2009 May 1;63(2):155-162
references.pmid: 22544972
references.citation: Med Care. 2000 Jun;38(6):552-61
references.pmid: 10843308
references.citation: Stat Med. 1997 Dec 15;16(23):2645-64
references.pmid: 9421867
references.citation: Stat Med. 2007 Apr 30;26(9):2088-112
references.pmid: 16906554
references.citation: Psychiatr Serv. 2009 Jan;60(1):67-73
references.pmid: 19114573
references.citation: Ann Intern Med. 1997 Oct 15;127(8 Pt 2):764-8
references.pmid: 9382395
references.citation: Health Serv Res. 2006 Aug;41(4 Pt 1):1413-37
references.pmid: 16899015