title: Automatic evidence quality prediction to support evidence-based decision making.
abstract: BACKGROUND
  Evidence-based medicine practice requires practitioners to obtain the best available medical evidence, and appraise the quality of the evidence when making clinical decisions. Primarily due to the plethora of electronically available data from the medical literature, the manual appraisal of the quality of evidence is a time-consuming process. We present a fully automatic approach for predicting the quality of medical evidence in order to aid practitioners at point-of-care.
  
  
  METHODS
  Our approach extracts relevant information from medical article abstracts and utilises data from a specialised corpus to apply supervised machine learning for the prediction of the quality grades. Following an in-depth analysis of the usefulness of features (e.g., publication types of articles), they are extracted from the text via rule-based approaches and from the meta-data associated with the articles, and then applied in the supervised classification model. We propose the use of a highly scalable and portable approach using a sequence of high precision classifiers, and introduce a simple evaluation metric called average error distance (AED) that simplifies the comparison of systems. We also perform elaborate human evaluations to compare the performance of our system against human judgments.
  
  
  RESULTS
  We test and evaluate our approaches on a publicly available, specialised, annotated corpus containing 1132 evidence-based recommendations. Our rule-based approach performs exceptionally well at the automatic extraction of publication types of articles, with F-scores of up to 0.99 for high-quality publication types. For evidence quality classification, our approach obtains an accuracy of 63.84% and an AED of 0.271. The human evaluations show that the performance of our system, in terms of AED and accuracy, is comparable to the performance of humans on the same data.
  
  
  CONCLUSIONS
  The experiments suggest that our structured text classification framework achieves evaluation results comparable to those of human performance. Our overall classification approach and evaluation technique are also highly portable and can be used for various evidence grading scales.
authors.affiliation: Department of Computing, Macquarie University, Sydney, NSW 2109, Australia. Electronic address: abeed.sarker@asu.edu.
authors.forename: Abeed
authors.initials: A
authors.lastname: Sarker
authors.affiliation: Department of Computing, Macquarie University, Sydney, NSW 2109, Australia.
authors.forename: Diego
authors.initials: D
authors.lastname: Mollá
authors.affiliation: Commonwealth Scientific and Industrial Research Organisation, Crn Vimiera and Pembroke Roads, Marsfield, NSW 2122, Australia.
authors.forename: Cécile
authors.initials: C
authors.lastname: Paris
mesh_terms: D001331:Automation; D057225:Data Mining / Q000379:methods*; D020000:Decision Support Systems, Clinical; D003661:Decision Support Techniques; D019317:Evidence-Based Medicine / Q000145:classification*; D019983:Guideline Adherence; D006801:Humans; D007600:Judgment; D009323:Natural Language Processing; D010363:Pattern Recognition, Automated; D017410:Practice Guidelines as Topic; D011381:Programming Languages; D011787:Quality of Health Care
keywords: Automatic medical evidence classification; Automatic text classification; Decision support system; Evidence-based medicine; Medical natural language processing