Assessment of commercial NLP engines for medication information extraction from dictated clinical notes.
OBJECTIVE
We assessed the current state of commercial natural language processing (NLP) engines for their ability to extract medication information from textual clinical documents.


METHODS
Two thousand de-identified discharge summaries and family practice notes were submitted to four commercial NLP engines with the request to extract all medication information. The four sets of returned results were combined to create a comparison standard which was validated against a manual, physician-derived gold standard created from a subset of 100 reports. Once validated, the individual vendor results for medication names, strengths, route, and frequency were compared against this automated standard with precision, recall, and F measures calculated.


RESULTS
Compared with the manual, physician-derived gold standard, the automated standard was successful at accurately capturing medication names (F measure=93.2%), but performed less well with strength (85.3%) and route (80.3%), and relatively poorly with dosing frequency (48.3%). Moderate variability was seen in the strengths of the four vendors. The vendors performed better with the structured discharge summaries than with the clinic notes in an analysis comparing the two document types.


CONCLUSIONS
Although automated extraction may serve as the foundation for a manual review process, it is not ready to automate medication lists without human intervention.
MedQuist Inc., 235 High Street, Suite 213, Morgantown, WV 26505, USA. juggy@medquist.com
V
V
Jagannathan
Charles J
CJ
Mullett
James G
JG
Arbogast
Kevin A
KA
Halbritter
Deepthi
D
Yellapragada
Sushmitha
S
Regulapati
Pavani
P
Bandaru
D001331:Automation; D004346:Drug Information Services; D009323:Natural Language Processing