title: Comparing Medline citations using modified N-grams.
abstract: OBJECTIVE
  We aim to identify duplicate pairs of Medline citations, particularly when the documents are not identical but contain similar information.
  
  
  METHODS
  Duplicate pairs of citations are identified by comparing word n-grams in pairs of documents. N-grams are modified using two approaches which take account of the fact that the document may have been altered. These are: (1) deletion, an item in the n-gram is removed; and (2) substitution, an item in the n-gram is substituted with a similar term obtained from the Unified Medical Language System Metathesaurus. N-grams are also weighted using a score derived from a language model. Evaluation is carried out using a set of 520 Medline citation pairs, including a set of 260 manually verified duplicate pairs obtained from the Deja Vu database.
  
  
  RESULTS
  The approach accurately detects duplicate Medline document pairs with an F1 measure score of 0.99. Allowing for word deletions and substitution improves performance. The best results are obtained by combining scores for n-grams of length 1-5 words.
  
  
  CONCLUSIONS
  Results show that the detection of duplicate Medline citations can be improved by modifying n-grams and that high performance can also be obtained using only unigrams (F1=0.959), particularly when allowing for substitutions of alternative phrases.
authors.affiliation: Department of Computer Science, COMSATS Institute of Information Technology, Lahore, Pakistan.
authors.forename: Rao Muhammad Adeel
authors.initials: RM
authors.lastname: Nawab
authors.forename: Mark
authors.initials: M
authors.lastname: Stevenson
authors.forename: Paul
authors.initials: P
authors.lastname: Clough
mesh_terms: D015713:Duplicate Publications as Topic; D016239:MEDLINE; D055641:Mathematical Concepts; D015233:Models, Statistical; D015714:Plagiarism; D017432:Unified Medical Language System
keywords: Natural Language Processing; PubMed