title: Evaluating the Quantitative Capabilities of Metagenomic Analysis Software.
abstract: DNA sequencing technologies are applied widely and frequently today to describe metagenomes, i.e., microbial communities in environmental or clinical samples, without the need for culturing them. These technologies usually return short (100-300 base-pairs long) DNA reads, and these reads are processed by metagenomic analysis software that assign phylogenetic composition-information to the dataset. Here we evaluate three metagenomic analysis software (AmphoraNet--a webserver implementation of AMPHORA2--, MG-RAST, and MEGAN5) for their capabilities of assigning quantitative phylogenetic information for the data, describing the frequency of appearance of the microorganisms of the same taxa in the sample. The difficulties of the task arise from the fact that longer genomes produce more reads from the same organism than shorter genomes, and some software assign higher frequencies to species with longer genomes than to those with shorter ones. This phenomenon is called the "genome length bias." Dozens of complex artificial metagenome benchmarks can be found in the literature. Because of the complexity of those benchmarks, it is usually difficult to judge the resistance of a metagenomic software to this "genome length bias." Therefore, we have made a simple benchmark for the evaluation of the "taxon-counting" in a metagenomic sample: we have taken the same number of copies of three full bacterial genomes of different lengths, break them up randomly to short reads of average length of 150 bp, and mixed the reads, creating our simple benchmark. Because of its simplicity, the benchmark is not supposed to serve as a mock metagenome, but if a software fails on that simple task, it will surely fail on most real metagenomes. We applied three software for the benchmark. The ideal quantitative solution would assign the same proportion to the three bacterial taxa. We have found that AMPHORA2/AmphoraNet gave the most accurate results and the other two software were under-performers: they counted quite reliably each short read to their respective taxon, producing the typical genome length bias. The benchmark dataset is available at http://pitgroup.org/static/3RandomGenome-100kavg150bps.fna.
authors.affiliation: PIT Bioinformatics Group, Eötvös University, Pázmány Péter stny. 1/C, Budapest, 1117, Hungary.
authors.forename: Csaba
authors.initials: C
authors.lastname: Kerepesi
authors.affiliation: PIT Bioinformatics Group, Eötvös University, Pázmány Péter stny. 1/C, Budapest, 1117, Hungary. grolmusz@pitgroup.org.|Uratim Ltd., Budapest, 1118, Hungary. grolmusz@pitgroup.org.
authors.forename: Vince
authors.initials: V
authors.lastname: Grolmusz
mesh_terms: D001419:Bacteria / Q000145:classification / Q000235:genetics* / Q000302:isolation & purification*; D016680:Genome, Bacterial; D056186:Metagenomics / Q000295:instrumentation / Q000379:methods*; D017422:Sequence Analysis, DNA; D012984:Software
keywords: Environmental microbiology; Genomics; Microbiology
references.citation: Microb Ecol. 2015 Apr;69(3):695-7
references.pmid: 25296554
references.citation: Genome Res. 2011 Sep;21(9):1552-60
references.pmid: 21690186
references.citation: PLoS One. 2008 Oct 08;3(10):e3373
references.pmid: 18841204
references.citation: Bioinformatics. 2012 Apr 1;28(7):1033-4
references.pmid: 22332237
references.citation: Genome Biol. 2008 Oct 13;9(10):R151
references.pmid: 18851752
references.citation: Genome Res. 2007 Mar;17(3):377-86
references.pmid: 17255551
references.citation: BMC Bioinformatics. 2008 Sep 19;9:386
references.pmid: 18803844
references.citation: Gene. 2014 Jan 10;533(2):538-40
references.pmid: 24144838
references.citation: Nat Methods. 2007 Jun;4(6):495-500
references.pmid: 17468765