title: Machine learning to assist risk-of-bias assessments in systematic reviews.
abstract: BACKGROUND
  Risk-of-bias assessments are now a standard component of systematic reviews. At present, reviewers need to manually identify relevant parts of research articles for a set of methodological elements that affect the risk of bias, in order to make a risk-of-bias judgement for each of these elements. We investigate the use of text mining methods to automate risk-of-bias assessments in systematic reviews. We aim to identify relevant sentences within the text of included articles, to rank articles by risk of bias and to reduce the number of risk-of-bias assessments that the reviewers need to perform by hand.
  
  
  METHODS
  We use supervised machine learning to train two types of models, for each of the three risk-of-bias properties of sequence generation, allocation concealment and blinding. The first model predicts whether a sentence in a research article contains relevant information. The second model predicts a risk-of-bias value for each research article. We use logistic regression, where each independent variable is the frequency of a word in a sentence or article, respectively.
  
  
  RESULTS
  We found that sentences can be successfully ranked by relevance with area under the receiver operating characteristic (ROC) curve (AUC) > 0.98. Articles can be ranked by risk of bias with AUC > 0.72. We estimate that more than 33% of articles can be assessed by just one reviewer, where two reviewers are normally required.
  
  
  CONCLUSIONS
  We show that text mining can be used to assist risk-of-bias assessments.
authors.affiliation: MRC Integrative Epidemiology Unit, School of Social and Community Medicine and Intelligent Systems Laboratory, University of Bristol, Bristol, UK louise.millard@bristol.ac.uk.
authors.forename: Louise A C
authors.initials: LA
authors.lastname: Millard
authors.affiliation: MRC Integrative Epidemiology Unit, Intelligent Systems Laboratory, University of Bristol, Bristol, UK.
authors.forename: Peter A
authors.initials: PA
authors.lastname: Flach
authors.affiliation: MRC Integrative Epidemiology Unit, School of Social and Community Medicine and.
authors.forename: Julian P T
authors.initials: JP
authors.lastname: Higgins
mesh_terms: D015982:Bias; D057225:Data Mining / Q000379:methods*; D066264:Datasets as Topic; D006801:Humans; D016015:Logistic Models; D000069550:Machine Learning / Q000706:statistics & numerical data*; D012372:ROC Curve; D012196:Review Literature as Topic
keywords: Risk of bias; machine learning; systematic review; text mining