The evaluation of a temporal reasoning system in processing clinical discharge summaries.
BACKGROUND
TimeText is a temporal reasoning system designed to represent, extract, and reason about temporal information in clinical text.


OBJECTIVE
To measure the accuracy of the TimeText for processing clinical discharge summaries.


METHODS
Six physicians with biomedical informatics training served as domain experts. Twenty discharge summaries were randomly selected for the evaluation. For each of the first 14 reports, 5 to 8 clinically important medical events were chosen. The temporal reasoning system generated temporal relations about the endpoints (start or finish) of pairs of medical events. Two experts (subjects) manually generated temporal relations for these medical events. The system and expert-generated results were assessed by four other experts (raters). All of the twenty discharge summaries were used to assess the system's accuracy in answering time-oriented clinical questions. For each report, five to ten clinically plausible temporal questions about events were generated. Two experts generated answers to the questions to serve as the gold standard. We wrote queries to retrieve answers from system's output.


METHODS
Correctness of generated temporal relations, recall of clinically important relations, and accuracy in answering temporal questions.


RESULTS
The raters determined that 97% of subjects' 295 generated temporal relations were correct and that 96.5% of the system's 995 generated temporal relations were correct. The system captured 79% of 307 temporal relations determined to be clinically important by the subjects and raters. The system answered 84% of the temporal questions correctly.


CONCLUSIONS
The system encoded the majority of information identified by experts, and was able to answer simple temporal questions.
Zhou
Li
L
Department of Biomedical Informatics, Columbia University, New York, NY, USA. lzhou2@partners.org
Parsons
Simon
S
Hripcsak
George
G
D005568:Forms and Records Control; D006801:Humans; D016247:Information Storage and Retrieval; D016347:Medical Records Systems, Computerized; D009323:Natural Language Processing; D010351:Patient Discharge; D013995:Time