title: The doubly conditioned frequency spectrum does not distinguish between ancient population structure and hybridization.
abstract: Distinguishing between hybridization and population structure in the ancestral species is a key challenge in our understanding of how permeable species boundaries are to gene flow. The doubly conditioned frequency spectrum (dcfs) has been argued to be a powerful metric to discriminate between these two explanations, and it was used to argue for hybridization between Neandertal and anatomically modern humans. The shape of the observed dcfs for these two species cannot be reproduced by a model that represents ancient population structure in Africa with two populations, while adding hybridization produces realistic shapes. In this letter, we show that this result is a consequence of the spatial coarseness of the demographic model and that a spatially structured stepping stone model can generate realistic dcfs without hybridization. This result highlights how inferences on hybridization between recently diverged species can be strongly affected by the choice of how population structure is represented in the underlying demographic model. We also conclude that the dcfs has limited power in distinguishing between the signals left by hybridization and ancient structure.
authors.affiliation: Evolutionary Ecology Group, Department of Zoology, University of Cambridge, Cambridge, United KingdomIntegrative Systems Biology Laboratory, King Abdullah University of Science and Technology (KAUST), Thuwal 23955-6900, Kingdom of Saudi Arabia aje44@cam.ac.uk am315@cam.ac.uk.
authors.forename: Anders
authors.initials: A
authors.lastname: Eriksson
authors.affiliation: Evolutionary Ecology Group, Department of Zoology, University of Cambridge, Cambridge, United Kingdom aje44@cam.ac.uk am315@cam.ac.uk.
authors.forename: Andrea
authors.initials: A
authors.lastname: Manica
mesh_terms: D000818:Animals; D044383:Black People / Q000235:genetics*; D003710:Demography; D051456:Gene Flow; D014644:Genetic Variation; D006801:Humans; D006824:Hybridization, Genetic; D008957:Models, Genetic; D059125:Neanderthals / Q000235:genetics*; D010802:Phylogeny
keywords: Neandertal; demography; hybridization; population structure