title: Artificial and natural duplicates in pyrosequencing reads of metagenomic data.
abstract: BACKGROUND
  Artificial duplicates from pyrosequencing reads may lead to incorrect interpretation of the abundance of species and genes in metagenomic studies. Duplicated reads were filtered out in many metagenomic projects. However, since the duplicated reads observed in a pyrosequencing run also include natural (non-artificial) duplicates, simply removing all duplicates may also cause underestimation of abundance associated with natural duplicates.
  
  
  RESULTS
  We implemented a method for identification of exact and nearly identical duplicates from pyrosequencing reads. This method performs an all-against-all sequence comparison and clusters the duplicates into groups using an algorithm modified from our previous sequence clustering method cd-hit. This method can process a typical dataset in approximately 10 minutes; it also provides a consensus sequence for each group of duplicates. We applied this method to the underlying raw reads of 39 genomic projects and 10 metagenomic projects that utilized pyrosequencing technique. We compared the occurrences of the duplicates identified by our method and the natural duplicates made by independent simulations. We observed that the duplicates, including both artificial and natural duplicates, make up 4-44% of reads. The number of natural duplicates highly correlates with the samples' read density (number of reads divided by genome size). For high-complexity metagenomic samples lacking dominant species, natural duplicates only make up <1% of all duplicates. But for some other samples like transcriptomic samples, majority of the observed duplicates might be natural duplicates.
  
  
  CONCLUSIONS
  Our method is available from http://cd-hit.org as a downloadable program and a web server. It is important not only to identify the duplicates from metagenomic datasets but also to distinguish whether they are artificial or natural duplicates. We provide a tool to estimate the number of natural duplicates according to user-defined sample types, so users can decide whether to retain or remove duplicates in their projects.
authors.affiliation: California Institute for Telecommunications and Information Technology, University of California San Diego, La Jolla, California 92093, USA.
authors.forename: Beifang
authors.initials: B
authors.lastname: Niu
authors.forename: Limin
authors.initials: L
authors.lastname: Fu
authors.forename: Shulei
authors.initials: S
authors.lastname: Sun
authors.forename: Weizhong
authors.initials: W
authors.lastname: Li
mesh_terms: D000465:Algorithms; D030541:Databases, Genetic; D016678:Genome; D020407:Internet; D056186:Metagenomics / Q000379:methods*; D017422:Sequence Analysis, DNA / Q000379:methods*; D012984:Software
references.citation: Science. 2004 Apr 2;304(5667):66-74
references.pmid: 15001713
references.citation: PLoS Biol. 2007 Mar;5(3):e77
references.pmid: 17355176
references.citation: Nature. 2004 Mar 4;428(6978):37-43
references.pmid: 14961025
references.citation: Science. 2006 Jan 27;311(5760):496-503
references.pmid: 16439655
references.citation: Environ Microbiol. 2009 Jun;11(6):1358-75
references.pmid: 19207571
references.citation: Nat Biotechnol. 2008 Oct;26(10):1135-45
references.pmid: 18846087
references.citation: Nature. 2009 Jan 22;457(7228):480-4
references.pmid: 19043404
references.citation: Genome Biol. 2007;8(7):R143
references.pmid: 17659080
references.citation: Science. 2005 Apr 22;308(5721):554-7
references.pmid: 15845853
references.citation: Nature. 2005 Sep 15;437(7057):376-80
references.pmid: 16056220
references.citation: Nat Methods. 2008 Feb;5(2):179-81
references.pmid: 18193056
references.citation: Nucleic Acids Res. 1994 Nov 11;22(22):4673-80
references.pmid: 7984417
references.citation: Science. 2006 Jun 2;312(5778):1355-9
references.pmid: 16741115
references.citation: ISME J. 2009 Nov;3(11):1314-7
references.pmid: 19587772
references.citation: Bioinformatics. 2006 Jul 1;22(13):1658-9
references.pmid: 16731699
references.citation: PLoS One. 2008 Aug 22;3(8):e3042
references.pmid: 18725995
references.citation: Nat Methods. 2007 Jun;4(6):495-500
references.pmid: 17468765
references.citation: Bioinformatics. 2001 Mar;17(3):282-3
references.pmid: 11294794
references.citation: Bioinformatics. 2002 Jan;18(1):77-82
references.pmid: 11836214
references.citation: Proc Natl Acad Sci U S A. 2008 Mar 11;105(10):3805-10
references.pmid: 18316740
references.citation: J Comput Biol. 2000 Feb-Apr;7(1-2):203-14
references.pmid: 10890397
references.citation: Nature. 2008 Apr 3;452(7187):629-32
references.pmid: 18337718