Measuring practice systems for chronic illness care: accuracy of self-reports from clinical personnel.
BACKGROUND
Purchasers, plans, and clinical practices involved in quality improvement initiatives are increasingly interested in measuring practice systems, particularly in relation to clinical quality and as part of pay-for-quality initiatives. The validity of self-reports of the use of practice systems was examined.


METHODS
In 11 medical groups in Minnesota, the Physician Practice Connections Readiness Survey, which was developed on the basis of the concepts and evidence base of the Chronic Care Model, was used to survey office practice personnel about practice systems. Participation rates by medical group ranged from 61% to 94%, with a mean of 76%, yielding surveys from 32 lead physicians and 241 other personnel. Survey results were compared with an on-site audit by trained surveyors.


RESULTS
Overall agreement with the on-site audit ranged from 40.9% to 96.7% among lead physicians and from 33.9% to 81.9% among other personnel. Mean agreement was high for quality improvement (96.7% for lead physicians and 81.9% for other personnel), moderate for clinical information systems (71.2% for lead physicians and 66.0% for others), and low for the use of care management (less than 50% for both groups). Mean positive predictive value ranged from 55.2% to 100% among lead physicians and from 49.6% to 100% among other personnel. Both the presence of systems and the accuracy of reporting varied across medical groups.


CONCLUSIONS
The accuracy of self-reports of practice systems varies by type of system being assessed and by type of respondent. Although self-assessment may be useful for quality improvement purposes, self-reported information on clinical practices systems should not be used for accountability purposes, including pay-for-quality efforts or public reporting unless additional documentation is required to ensure fair comparisons.
Scholle
Sarah Hudson
SH
National Committee for Quality Assurance, Washington, DC, USA. Scholle@ncqa.org
Pawlson
L Gregory
LG
Solberg
Leif I
LI
Shih
Sarah C
SC
Asche
Stephen E
SE
Chou
Ann F
AF
Thoele
Merry Jo
MJ
D019985:Benchmarking; D002908:Chronic Disease; D006123:Group Practice; D019538:Health Care Surveys; D006305:Health Status Indicators; D006801:Humans; D008485:Medical Audit; D008910:Minnesota; D010043:Outcome and Process Assessment, Health Care / Q000379:methods*; D010818:Practice Patterns, Physicians'