title: In the pursuit of a semantic similarity metric based on UMLS annotations for articles in PubMed Central Open Access.
abstract: BACKGROUND
  Although full-text articles are provided by the publishers in electronic formats, it remains a challenge to find related work beyond the title and abstract context. Identifying related articles based on their abstract is indeed a good starting point; this process is straightforward and does not consume as many resources as full-text based similarity would require. However, further analyses may require in-depth understanding of the full content. Two articles with highly related abstracts can be substantially different regarding the full content. How similarity differs when considering title-and-abstract versus full-text and which semantic similarity metric provides better results when dealing with full-text articles are the main issues addressed in this manuscript.
  
  
  METHODS
  We have benchmarked three similarity metrics - BM25, PMRA, and Cosine, in order to determine which one performs best when using concept-based annotations on full-text documents. We also evaluated variations in similarity values based on title-and-abstract against those relying on full-text. Our test dataset comprises the Genomics track article collection from the 2005 Text Retrieval Conference. Initially, we used an entity recognition software to semantically annotate titles and abstracts as well as full-text with concepts defined in the Unified Medical Language System (UMLS®). For each article, we created a document profile, i.e., a set of identified concepts, term frequency, and inverse document frequency; we then applied various similarity metrics to those document profiles. We considered correlation, precision, recall, and F1 in order to determine which similarity metric performs best with concept-based annotations. For those full-text articles available in PubMed Central Open Access (PMC-OA), we also performed dispersion analyses in order to understand how similarity varies when considering full-text articles.
  
  
  RESULTS
  We have found that the PubMed Related Articles similarity metric is the most suitable for full-text articles annotated with UMLS concepts. For similarity values above 0.8, all metrics exhibited an F1 around 0.2 and a recall around 0.1; BM25 showed the highest precision close to 1; in all cases the concept-based metrics performed better than the word-stem-based one. Our experiments show that similarity values vary when considering only title-and-abstract versus full-text similarity. Therefore, analyses based on full-text become useful when a given research requires going beyond title and abstract, particularly regarding connectivity across articles.
  
  
  BACKGROUND
  Visualization available at ljgarcia.github.io/semsim.benchmark/, data available at http://dx.doi.org/10.5281/zenodo.13323.
authors.affiliation: Temporal Knowledge Bases Group, Department of Computer Languages and Systems, Universitat Jaume I, 12071 Castelló de la Plana, Spain. Electronic address: al278693@uji.es.
authors.forename: Leyla Jael
authors.initials: LJ
authors.lastname: Garcia Castro
authors.affiliation: Temporal Knowledge Bases Group, Department of Computer Languages and Systems, Universitat Jaume I, 12071 Castelló de la Plana, Spain.
authors.forename: Rafael
authors.initials: R
authors.lastname: Berlanga
authors.affiliation: Ontology Engineering Group, Informatics Department, Universidad Politécnica de Madrid, 28660 Boadilla del Monte, Madrid, Spain; Linkingdata I/O LLC, Fort Collins, Colorado, USA.
authors.forename: Alexander
authors.initials: A
authors.lastname: Garcia
mesh_terms: D022126:Access to Information; D066289:Data Curation; D006801:Humans; D016247:Information Storage and Retrieval; D039781:PubMed; D012660:Semantics; D012984:Software; D017432:Unified Medical Language System
keywords: Related articles; Scientific publications; Semantic annotations; Semantic similarity; Similarity metrics