title: Testing for ancient admixture between closely related populations.
abstract: One enduring question in evolutionary biology is the extent of archaic admixture in the genomes of present-day populations. In this paper, we present a test for ancient admixture that exploits the asymmetry in the frequencies of the two nonconcordant gene trees in a three-population tree. This test was first applied to detect interbreeding between Neandertals and modern humans. We derive the analytic expectation of a test statistic, called the D statistic, which is sensitive to asymmetry under alternative demographic scenarios. We show that the D statistic is insensitive to some demographic assumptions such as ancestral population sizes and requires only the assumption that the ancestral populations were randomly mating. An important aspect of D statistics is that they can be used to detect archaic admixture even when no archaic sample is available. We explore the effect of sequencing error on the false-positive rate of the test for admixture, and we show how to estimate the proportion of archaic ancestry in the genomes of present-day populations. We also investigate a model of subdivision in ancestral populations that can result in D statistics that indicate recent admixture.
authors.affiliation: Department of Integrative Biology, University of California, Berkeley, CA, USA. eric.durand@berkeley.edu
authors.forename: Eric Y
authors.initials: EY
authors.lastname: Durand
authors.forename: Nick
authors.initials: N
authors.lastname: Patterson
authors.forename: David
authors.initials: D
authors.lastname: Reich
authors.forename: Montgomery
authors.initials: M
authors.lastname: Slatkin
mesh_terms: D000465:Algorithms; D000818:Animals; D003198:Computer Simulation; D019143:Evolution, Molecular; D051456:Gene Flow / Q000235:genetics; D014644:Genetic Variation / Q000235:genetics; D005828:Genetics, Population; D016678:Genome / Q000235:genetics; D006801:Humans; D015233:Models, Statistical