title: Automatic evidence retrieval for systematic reviews.
abstract: BACKGROUND
  Snowballing involves recursively pursuing relevant references cited in the retrieved literature and adding them to the search results. Snowballing is an alternative approach to discover additional evidence that was not retrieved through conventional search. Snowballing's effectiveness makes it best practice in systematic reviews despite being time-consuming and tedious.
  
  
  OBJECTIVE
  Our goal was to evaluate an automatic method for citation snowballing's capacity to identify and retrieve the full text and/or abstracts of cited articles.
  
  
  METHODS
  Using 20 review articles that contained 949 citations to journal or conference articles, we manually searched Microsoft Academic Search (MAS) and identified 78.0% (740/949) of the cited articles that were present in the database. We compared the performance of the automatic citation snowballing method against the results of this manual search, measuring precision, recall, and F1 score.
  
  
  RESULTS
  The automatic method was able to correctly identify 633 (as proportion of included citations: recall=66.7%, F1 score=79.3%; as proportion of citations in MAS: recall=85.5%, F1 score=91.2%) of citations with high precision (97.7%), and retrieved the full text or abstract for 490 (recall=82.9%, precision=92.1%, F1 score=87.3%) of the 633 correctly retrieved citations.
  
  
  CONCLUSIONS
  The proposed method for automatic citation snowballing is accurate and is capable of obtaining the full texts or abstracts for a substantial proportion of the scholarly citations in review articles. By automating the process of citation snowballing, it may be possible to reduce the time and effort of common evidence surveillance tasks such as keeping trial registries up to date and conducting systematic reviews.
authors.affiliation: Centre for Health Informatics, Australian Institute of Health Innovation, University of New South Wales, Kensington NSW, Australia.
authors.forename: Miew Keen
authors.identifier: http://orcid.org/0000-0003-0352-5585
authors.initials: MK
authors.lastname: Choong
authors.forename: Filippo
authors.identifier: http://orcid.org/0000-0002-3553-3611
authors.initials: F
authors.lastname: Galgani
authors.forename: Adam G
authors.identifier: http://orcid.org/0000-0002-1720-8209
authors.initials: AG
authors.lastname: Dunn
authors.forename: Guy
authors.identifier: http://orcid.org/0000-0003-4353-2026
authors.initials: G
authors.lastname: Tsafnat
mesh_terms: D016208:Databases, Factual; D019317:Evidence-Based Medicine; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*; D008490:Medical Informatics / Q000379:methods*; D012042:Registries; D012196:Review Literature as Topic
keywords: evidence-based medicine; information storage and retrieval; medical informatics