title: Reducing systematic review workload through certainty-based screening.
abstract: In systematic reviews, the growing number of published studies imposes a significant screening workload on reviewers. Active learning is a promising approach to reduce the workload by automating some of the screening decisions, but it has been evaluated for a limited number of disciplines. The suitability of applying active learning to complex topics in disciplines such as social science has not been studied, and the selection of useful criteria and enhancements to address the data imbalance problem in systematic reviews remains an open problem. We applied active learning with two criteria (certainty and uncertainty) and several enhancements in both clinical medicine and social science (specifically, public health) areas, and compared the results in both. The results show that the certainty criterion is useful for finding relevant documents, and weighting positive instances is promising to overcome the data imbalance problem in both data sets. Latent dirichlet allocation (LDA) is also shown to be promising when little manually-assigned information is available. Active learning is effective in complex topics, although its efficiency is limited due to the difficulties in text classification. The most promising criterion and weighting method are the same regardless of the review topic, and unsupervised techniques like LDA have a possibility to boost the performance of active learning without manual annotation.
authors: The National Centre for Text Mining and School of Computer Science, Manchester Institute of Biotechnology, University of Manchester, 131 Princess Street, Manchester M1 7DN, UK; Toyota Technological Institute, 2-12-1 Hisakata, Tempaku-ku, Nagoya 468-8511, Japan. Electronic address: makoto-miwa@toyota-ti.ac.jp.
authors: Makoto
authors: M
authors: Miwa
authors: Evidence for Policy and Practice Information and Coordinating (EPPI-)Centre, Social Science Research Unit, Institute of Education, University of London, London, UK. Electronic address: j.thomas@ioe.ac.uk.
authors: James
authors: J
authors: Thomas
authors: Evidence for Policy and Practice Information and Coordinating (EPPI-)Centre, Social Science Research Unit, Institute of Education, University of London, London, UK. Electronic address: a.omara-eves@ioe.ac.uk.
authors: Alison
authors: A
authors: O'Mara-Eves
authors: The National Centre for Text Mining and School of Computer Science, Manchester Institute of Biotechnology, University of Manchester, 131 Princess Street, Manchester M1 7DN, UK. Electronic address: Sophia.Ananiadou@manchester.ac.uk.
authors: Sophia
authors: S
authors: Ananiadou
mesh_terms: D000043:Abstracting and Indexing / Q000379:methods; D000465:Algorithms; D001185:Artificial Intelligence; D016206:Databases, Bibliographic / Q000145:classification; D008373:Manuscripts as Topic; D009323:Natural Language Processing; D017712:Peer Review, Research / Q000379:methods; D012660:Semantics; D016526:Workload; D000078202:Systematic Reviews as Topic
keywords: Active learning; Certainty; Systematic reviews; Text mining