title: Two-Locus Likelihoods Under Variable Population Size and Fine-Scale Recombination Rate Estimation.
abstract: Two-locus sampling probabilities have played a central role in devising an efficient composite-likelihood method for estimating fine-scale recombination rates. Due to mathematical and computational challenges, these sampling probabilities are typically computed under the unrealistic assumption of a constant population size, and simulation studies have shown that resulting recombination rate estimates can be severely biased in certain cases of historical population size changes. To alleviate this problem, we develop here new methods to compute the sampling probability for variable population size functions that are piecewise constant. Our main theoretical result, implemented in a new software package called LDpop, is a novel formula for the sampling probability that can be evaluated by numerically exponentiating a large but sparse matrix. This formula can handle moderate sample sizes ([Formula: see text]) and demographic size histories with a large number of epochs ([Formula: see text]). In addition, LDpop implements an approximate formula for the sampling probability that is reasonably accurate and scales to hundreds in sample size ([Formula: see text]). Finally, LDpop includes an importance sampler for the posterior distribution of two-locus genealogies, based on a new result for the optimal proposal distribution in the variable-size setting. Using our methods, we study how a sharp population bottleneck followed by rapid growth affects the correlation between partially linked sites. Then, through an extensive simulation study, we show that accounting for population size changes under such a demographic model leads to substantial improvements in fine-scale recombination rate estimation.
authors.affiliation: Department of Statistics, University of California, Berkeley, California 94720 Computer Science Division, University of California, Berkeley, California 94720.
authors.forename: John A
authors.initials: JA
authors.lastname: Kamm
authors.affiliation: Computational Biology Graduate Group, University of California, Berkeley, California 94720.
authors.forename: Jeffrey P
authors.initials: JP
authors.lastname: Spence
authors.affiliation: Computer Science Division, University of California, Berkeley, California 94720.
authors.forename: Jeffrey
authors.initials: J
authors.lastname: Chan
authors.affiliation: Department of Statistics, University of California, Berkeley, California 94720 Computer Science Division, University of California, Berkeley, California 94720 Department of Integrative Biology, University of California, Berkeley, California 94720 Departments of Mathematics and Biology, University of Pennsylvania, Philadelphia, Pennsylvania 19104 yss@berkeley.edu.
authors.forename: Yun S
authors.initials: YS
authors.lastname: Song
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D005828:Genetics, Population; D006801:Humans; D016013:Likelihood Functions; D008957:Models, Genetic; D011156:Population Density; D011995:Recombination, Genetic
keywords: coalescent with recombination; importance sampling; sampling probability; two-locus Moran model