Natural language processing in the electronic medical record: assessing clinician adherence to tobacco treatment guidelines.
BACKGROUND
Comprehensively assessing care quality with electronic medical records (EMRs) is not currently possible because much data reside in clinicians' free-text notes.


METHODS
We evaluated the accuracy of MediClass, an automated, rule-based classifier of the EMR that incorporates natural language processing, in assessing whether clinicians: (1) asked if the patient smoked; (2) advised them to stop; (3) assessed their readiness to quit; (4) assisted them in quitting by providing information or medications; and (5) arranged for appropriate follow-up care (i.e., the 5A's of smoking-cessation care).


METHODS
We analyzed 125 medical records of known smokers at each of four HMOs in 2003 and 2004. One trained abstractor at each HMO manually coded all 500 records according to whether or not each of the 5A's of smoking cessation care was addressed during routine outpatient visits.


METHODS
For each patient's record, we compared the presence or absence of each of the 5A's as assessed by each human coder and by MediClass. We measured the chance-corrected agreement between the human raters and MediClass using the kappa statistic.


RESULTS
For "ask" and "assist," agreement among human coders was indistinguishable from agreement between humans and MediClass (p>0.05). For "assess" and "advise," the human coders agreed more with each other than they did with MediClass (p<0.01); however, MediClass performance was sufficient to assess quality in these areas. The frequency of "arrange" was too low to be analyzed.


CONCLUSIONS
MediClass performance appears adequate to replace human coders of the 5A's of smoking-cessation care, allowing for automated assessment of clinician adherence to one of the most important, evidence-based guidelines in preventive health care.
Hazlehurst
Brian
B
Center for Health Research, Kaiser Permanente, Portland, Oregon 97227, USA. brian.hazlehurst@kpchr.org
Sittig
Dean F
DF
Stevens
Victor J
VJ
Smith
K Sabina
KS
Hollis
Jack F
JF
Vogt
Thomas M
TM
Winickoff
Jonathan P
JP
Glasgow
Russ
R
Palen
Ted E
TE
Rigotti
Nancy A
NA
D003376:Counseling / Q000379:methods*; D017408:Guidelines as Topic; D008485:Medical Audit; D016347:Medical Records Systems, Computerized; D009323:Natural Language Processing; D010818:Practice Patterns, Physicians' / Q000592:standards*; D016540:Smoking Cessation; D012985:Software Design; D014481:United States