title: Automatic endpoint detection to support the systematic review process.
abstract: Preparing a systematic review can take hundreds of hours to complete, but the process of reconciling different results from multiple studies is the bedrock of evidence-based medicine. We introduce a two-step approach to automatically extract three facets - two entities (the agent and object) and the way in which the entities are compared (the endpoint) - from direct comparative sentences in full-text articles. The system does not require a user to predefine entities in advance and thus can be used in domains where entity recognition is difficult or unavailable. As with a systematic review, the tabular summary produced using the automatically extracted facets shows how experimental results differ between studies. Experiments were conducted using a collection of more than 2million sentences from three journals Diabetes, Carcinogenesis and Endocrinology and two machine learning algorithms, support vector machines (SVM) and a general linear model (GLM). F1 and accuracy measures for the SVM and GLM differed by only 0.01 across all three comparison facets in a randomly selected set of test sentences. The system achieved the best performance of 92% for objects, whereas the accuracy for both agent and endpoints was 73%. F1 scores were higher for objects (0.77) than for endpoints (0.51) or agents (0.47). A situated evaluation of Metformin, a drug to treat diabetes, showed system accuracy of 95%, 83% and 79% for the object, endpoint and agent respectively. The situated evaluation had higher F1 scores of 0.88, 0.64 and 0.62 for object, endpoint, and agent respectively. On average, only 5.31% of the sentences in a full-text article are direct comparisons, but the tabular summaries suggest that these sentences provide a rich source of currently underutilized information that can be used to accelerate the systematic review process and identify gaps where future research should be focused.
authors.affiliation: Graduate School of Library and Information Science and Medical Information Science, Center for Informatics Research in Science and Scholarship (CIRSS), University of Illinois, Urbana Champaign, 501 E. Daniel Street, MC-493, Champaign, IL 61820-6211, USA. Electronic address: clblake@illinois.edu.
authors.forename: Catherine
authors.initials: C
authors.lastname: Blake
authors.affiliation: Graduate School of Library and Information Science, University of Illinois, Urbana Champaign, 501 E. Daniel Street, MC-493, Champaign, IL 61820-6211, USA. Electronic address: alucic2@illinois.edu.
authors.forename: Ana
authors.initials: A
authors.lastname: Lucic
mesh_terms: D005260:Female; D006801:Humans; D000465:Algorithms; D001786:Blood Glucose / Q000032:analysis; D057186:Comparative Effectiveness Research; D057225:Data Mining / Q000379:methods; D001330:Electronic Data Processing; D019317:Evidence-Based Medicine; D007328:Insulin / Q000737:chemistry; D007802:Language; D016014:Linear Models; D008037:Linguistics; D000069550:Machine Learning; D008687:Metformin / Q000627:therapeutic use; D009323:Natural Language Processing; D020849:Raloxifene Hydrochloride / Q000627:therapeutic use; D016032:Randomized Controlled Trials as Topic; D015203:Reproducibility of Results; D060388:Support Vector Machine; D013629:Tamoxifen / Q000031:analogs & derivatives / Q000627:therapeutic use; D014599:Uterus / Q000187:drug effects; D000078202:Systematic Reviews as Topic
chemical_list: C016601:afimoxifene; D001786:Blood Glucose; D007328:Insulin; D008687:Metformin; D020849:Raloxifene Hydrochloride; D013629:Tamoxifen
keywords: Evidence-based medicine; Information extraction; Systematic review; Text mining