title: Identifying duplicate content using statistically improbable phrases.
abstract: BACKGROUND
  Document similarity metrics such as PubMed's 'Find related articles' feature, which have been primarily used to identify studies with similar topics, can now also be used to detect duplicated or potentially plagiarized papers within literature reference databases. However, the CPU-intensive nature of document comparison has limited MEDLINE text similarity studies to the comparison of abstracts, which constitute only a small fraction of a publication's total text. Extending searches to include text archived by online search engines would drastically increase comparison ability. For large-scale studies, submitting short phrases encased in direct quotes to search engines for exact matches would be optimal for both individual queries and programmatic interfaces. We have derived a method of analyzing statistically improbable phrases (SIPs) for assistance in identifying duplicate content.
  
  
  RESULTS
  When applied to MEDLINE citations, this method substantially improves upon previous algorithms in the detection of duplication citations, yielding a precision and recall of 78.9% (versus 50.3% for eTBLAST) and 99.6% (versus 99.8% for eTBLAST), respectively.
  
  
  BACKGROUND
  Similar citations identified by this work are freely accessible in the Déjà vu database, under the SIP discovery method category at http://dejavu.vbi.vt.edu/dejavu/.
authors.affiliation: Division of Translational Research, The University of Texas Southwestern Medical Center, 5323 Harry Hines Boulevard, Dallas, TX 75290-9185, USA. merrami@collin.edu
authors.forename: Mounir
authors.initials: M
authors.lastname: Errami
authors.forename: Zhaohui
authors.initials: Z
authors.lastname: Sun
authors.forename: Angela C
authors.initials: AC
authors.lastname: George
authors.forename: Tara C
authors.initials: TC
authors.lastname: Long
authors.forename: Michael A
authors.initials: MA
authors.lastname: Skinner
authors.forename: Jonathan D
authors.initials: JD
authors.lastname: Wren
authors.forename: Harold R
authors.initials: HR
authors.lastname: Garner
mesh_terms: D000043:Abstracting and Indexing / Q000379:methods*; D016208:Databases, Factual; D015713:Duplicate Publications as Topic; D016239:MEDLINE; D015714:Plagiarism; D039781:PubMed; D057386:Search Engine; D014481:United States
references.citation: Nucleic Acids Res. 2009 Jan;37(Database issue):D921-4
references.pmid: 18757888
references.citation: Otolaryngol Head Neck Surg. 2002 Mar;126(3):211-6
references.pmid: 11956527
references.citation: Science. 2009 Mar 6;323(5919):1293-4
references.pmid: 19265004
references.citation: Image J Nurs Sch. 1995 Spring;27(1):51-6
references.pmid: 7721317
references.citation: Nature. 2005 Jun 9;435(7043):737-8
references.pmid: 15944677
references.citation: JAMA. 2004 Feb 25;291(8):974-80
references.pmid: 14982913
references.citation: Ned Tijdschr Geneeskd. 1999 Oct 23;143(43):2150-3
references.pmid: 10568328
references.citation: Ophthalmology. 2004 May;111(5):863-6
references.pmid: 15121360
references.citation: J Plast Reconstr Aesthet Surg. 2006;59(9):975-7
references.pmid: 16920591
references.citation: Psychol Rep. 2005 Aug;97(1):43-9
references.pmid: 16279303
references.citation: Sci Eng Ethics. 2006 Jul;12(3):543-54
references.pmid: 16909156
references.citation: J Hand Surg Br. 2004 Dec;29(6):625-8
references.pmid: 15542228
references.citation: Nature. 2008 Jan 24;451(7177):397-9
references.pmid: 18216832
references.citation: Nucleic Acids Res. 2007 Jul;35(Web Server issue):W12-5
references.pmid: 17452348
references.citation: Eur J Clin Pharmacol. 1989;36(5):429-32
references.pmid: 2666138
references.citation: Surgery. 2001 Jun;129(6):655-61
references.pmid: 11391360
references.citation: Bioinformatics. 2008 Jan 15;24(2):243-9
references.pmid: 18056062
references.citation: Laryngoscope. 2003 May;113(5):772-4
references.pmid: 12792309
references.citation: Ned Tijdschr Geneeskd. 1993 Mar 20;137(12):593-7
references.pmid: 8459858