Using a stakeholder-engaged approach to develop and validate electronic clinical quality measures.
OBJECTIVE
To describe the stakeholder-engaged processes used to develop, specify, and validate 2 oral health care electronic clinical quality measures.


METHODS
A broad range of stakeholders were engaged from conception through testing to develop measures and test feasibility, reliability, and validity following National Quality Forum guidance. We assessed data element feasibility through semistructured interviews with key stakeholders using a National Quality Forum-recommended scorecard. We created test datasets of synthetic patients to test measure implementation feasibility and reliability within and across electronic health record (EHR) systems. We validated implementation with automated reporting of EHR clinical data against manual record reviews, using the kappa statistic.


RESULTS
A stakeholder workgroup was formed and guided all development and testing processes. All critical data elements passed feasibility testing. Four test datasets, representing 577 synthetic patients, were developed and implemented within EHR vendors' software, demonstrating measure implementation feasibility. Measure reliability and validity were established through implementation at clinical practice sites, with kappa statistic values in the "almost perfect" agreement range of 0.80-0.99 for all but 1 measure component, which demonstrated "substantial" agreement. The 2 validated measures were published in the United States Health Information Knowledgebase.


CONCLUSIONS
The stakeholder-engaged processes used in this study facilitated a successful measure development and testing cycle. Engaging stakeholders early and throughout development and testing promotes early identification of and attention to potential threats to feasibility, reliability, and validity, thereby averting significant resource investments that are unlikely to be fruitful.
Key Analytics and Consulting, LLC, Gainesville, Florida, USA (Dr Herndon was with the Department of Health Outcomes and Policy, University of Florida College of Medicine, Gainesville, Florida, USA, when this study was conducted).
Jill Boylston
JB
Herndon
Dental Quality Alliance, American Dental Association, Chicago, Illinois, USA.
Krishna
K
Aravamudhan
Stephenson Developers Group, Clearwater, Florida, USA.
Ronald L
RL
Stephenson
Exan Group, Vancouver, Canada.
Ryan
R
Brandon
American Dental Partners, Inc., Wakefield, Massachusetts, USA.
Jesley
J
Ruff
Department of Community Dentistry and Behavioral Science, University of Florida College of Dentistry, Gainesville, Florida, USA.
Frank
F
Catalanotto
Asian Health Services, Oakland, California and National Network for Oral Health Access, Denver, Colorado, USA.
Huong
H
Le
D000293:Adolescent; D002648:Child; D002675:Child, Preschool; D066264:Datasets as Topic; D003731:Dental Caries / Q000628:therapy; D057286:Electronic Health Records; D055094:Evidence-Based Dentistry; D006801:Humans; D062527:Meaningful Use; D010377:Pediatric Dentistry / Q000592:standards*; D019984:Quality Indicators, Health Care; D014481:United States; D055815:Young Adult
electronic health records; health care quality indicators; meaningful use; quality measure development; quality measure testing