title: Large-scale benchmarking reveals false discoveries and count transformation sensitivity in 16S rRNA gene amplicon data analysis methods used in microbiome studies.
abstract: There is an immense scientific interest in the human microbiome and its effects on human physiology, health, and disease. A common approach for examining bacterial communities is high-throughput sequencing of 16S rRNA gene hypervariable regions, aggregating sequence-similar amplicons into operational taxonomic units (OTUs). Strategies for detecting differential relative abundance of OTUs between sample conditions include classical statistical approaches as well as a plethora of newer methods, many borrowing from the related field of RNA-seq analysis. This effort is complicated by unique data characteristics, including sparsity, sequencing depth variation, and nonconformity of read counts to theoretical distributions, which is often exacerbated by exploratory and/or unbalanced study designs. Here, we assess the robustness of available methods for (1) inference in differential relative abundance analysis and (2) beta-diversity-based sample separation, using a rigorous benchmarking framework based on large clinical 16S microbiome datasets from different sources.
  
  
  
  Running more than 380,000 full differential relative abundance tests on real datasets with permuted case/control assignments and in silico-spiked OTUs, we identify large differences in method performance on a range of parameters, including false positive rates, sensitivity to sparsity and case/control balances, and spike-in retrieval rate. In large datasets, methods with the highest false positive rates also tend to have the best detection power. For beta-diversity-based sample separation, we show that library size normalization has very little effect and that the distance metric is the most important factor in terms of separation power.
  
  
  
  Our results, generalizable to datasets from different sequencing platforms, demonstrate how the choice of method considerably affects analysis outcome. Here, we give recommendations for tools that exhibit low false positive rates, have good retrieval power across effect sizes and case/control proportions, and have low sparsity bias. Result output from some commonly used methods should be interpreted with caution. We provide an easily extensible framework for benchmarking of new methods and future microbiome datasets.
authors.affiliation: COPSAC, Copenhagen Prospective Studies on Asthma in Childhood, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Jonathan
authors.initials: J
authors.lastname: Thorsen
authors.affiliation: Section of Microbiology, Department of Biology, University of Copenhagen, Copenhagen, Denmark.|Department of Biology, Laboratory of Genomics and Molecular Biomedicine, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Asker
authors.initials: A
authors.lastname: Brejnrod
authors.affiliation: Section of Microbiology, Department of Biology, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Martin
authors.initials: M
authors.lastname: Mortensen
authors.affiliation: COPSAC, Copenhagen Prospective Studies on Asthma in Childhood, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Morten A
authors.initials: MA
authors.lastname: Rasmussen
authors.affiliation: COPSAC, Copenhagen Prospective Studies on Asthma in Childhood, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Jakob
authors.initials: J
authors.lastname: Stokholm
authors.affiliation: Section of Microbiology, Department of Biology, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Waleed Abu
authors.initials: WA
authors.lastname: Al-Soud
authors.affiliation: Section of Microbiology, Department of Biology, University of Copenhagen, Copenhagen, Denmark.
authors.forename: Søren
authors.initials: S
authors.lastname: Sørensen
authors.affiliation: COPSAC, Copenhagen Prospective Studies on Asthma in Childhood, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark. bisgaard@copsac.com.
authors.forename: Hans
authors.initials: H
authors.lastname: Bisgaard
authors.affiliation: COPSAC, Copenhagen Prospective Studies on Asthma in Childhood, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark. johannes.waage@dbac.dk.
authors.forename: Johannes
authors.identifier: 0000-0002-0603-8822
authors.initials: J
authors.lastname: Waage
mesh_terms: D001419:Bacteria / Q000145:classification* / Q000235:genetics*; D001483:Base Sequence; D019985:Benchmarking; D016022:Case-Control Studies; D019295:Computational Biology / Q000379:methods; D005189:False Positive Reactions; D059014:High-Throughput Nucleotide Sequencing / Q000379:methods*; D006801:Humans; D064307:Microbiota / Q000235:genetics*; D012336:RNA, Ribosomal, 16S / Q000235:genetics*; D017423:Sequence Analysis, RNA / Q000379:methods
chemical_list: D012336:RNA, Ribosomal, 16S
keywords: 16S sequencing; Benchmark; Beta-diversity; Differential relative abundance; Microbiome