Using Machine Learning and Natural Language Processing Algorithms to Automate the Evaluation of Clinical Decision Support in Electronic Medical Record Systems.
BACKGROUND
As the number of clinical decision support systems (CDSSs) incorporated into electronic medical records (EMRs) increases, so does the need to evaluate their effectiveness. The use of medical record review and similar manual methods for evaluating decision rules is laborious and inefficient. The authors use machine learning and Natural Language Processing (NLP) algorithms to accurately evaluate a clinical decision support rule through an EMR system, and they compare it against manual evaluation.


METHODS
Modeled after the EMR system EPIC at Maine Medical Center, we developed a dummy data set containing physician notes in free text for 3,621 artificial patients records undergoing a head computed tomography (CT) scan for mild traumatic brain injury after the incorporation of an electronic best practice approach. We validated the accuracy of the Best Practice Advisories (BPA) using three machine learning algorithms-C-Support Vector Classification (SVC), Decision Tree Classifier (DecisionTreeClassifier), k-nearest neighbors classifier (KNeighborsClassifier)-by comparing their accuracy for adjudicating the occurrence of a mild traumatic brain injury against manual review. We then used the best of the three algorithms to evaluate the effectiveness of the BPA, and we compared the algorithm's evaluation of the BPA to that of manual review.


RESULTS
The electronic best practice approach was found to have a sensitivity of 98.8 percent (96.83-100.0), specificity of 10.3 percent, PPV = 7.3 percent, and NPV = 99.2 percent when reviewed manually by abstractors. Though all the machine learning algorithms were observed to have a high level of prediction, the SVC displayed the highest with a sensitivity 93.33 percent (92.49-98.84), specificity of 97.62 percent (96.53-98.38), PPV = 50.00, NPV = 99.83. The SVC algorithm was observed to have a sensitivity of 97.9 percent (94.7-99.86), specificity 10.30 percent, PPV 7.25 percent, and NPV 99.2 percent for evaluating the best practice approach, after accounting for 17 cases (0.66 percent) where the patient records had to be reviewed manually due to the NPL systems inability to capture the proper diagnosis.


CONCLUSIONS
CDSSs incorporated into EMRs can be evaluated in an automatic fashion by using NLP and machine learning techniques.
Muskie School of Public Service, University of Southern Maine.
Donald A
DA
Szlosek
Champlain College.
Jonathan
J
Ferrett
classification; data collection; data use and quality; electronic health records; health information technology; machine learning; natural language processing; quality improvement
Ann Emerg Med. 2013 May;61(5):521-7
22921164
JAMA. 2005 Sep 28;294(12 ):1519-25
16189365
Pharmacoepidemiol Drug Saf. 2015 Jan;24(1):86-92
25402257
ACP J Club. 2006 Mar-Apr;144(2):53
16539368
Ann Emerg Med. 2001 Sep;38(3):317-22
11524653
JAMA. 2005 Sep 28;294(12 ):1511-8
16189364
Acad Emerg Med. 2012 Jan;19(1):2-10
22251188
PLoS One. 2015 Aug 24;10(8):e0136651
26301417
BMC Health Serv Res. 2003 Jun 06;3(1):10
12793909
Evid Based Med. 2006 Apr;11(2):61
17213091
Pediatr Radiol. 2011 Aug;41(8):971-9
21465153
Radiology. 2007 Dec;245(3):831-8
17911536
World J Emerg Surg. 2014 Apr 17;9:31
24742359