title: An integrated-likelihood method for estimating genetic differentiation between populations.
abstract: The aim of this article is to develop an integrated-likelihood (IL) approach to estimate the genetic differentiation between populations. The conventional maximum-likelihood (ML) and pseudolikelihood (PL) methods that use sample counts of alleles may cause severe underestimations of FST, which means overestimations of theta=4Nm, when the number of sampling localities is small. To reduce such bias in the estimation of genetic differentiation, we propose an IL method in which the mean allele frequencies over populations are regarded as nuisance parameters and are eliminated by integration. To maximize the IL function, we have developed two algorithms, a Monte Carlo EM algorithm and a Laplace approximation. Our simulation studies show that the method proposed here outperforms the conventional ML and PL methods in terms of unbiasedness and precision. The IL method was applied to real data for Pacific herring and African elephants.
authors.affiliation: Faculty of Marine Science, Tokyo University of Marine Science and Technology, Japan. kitakado@s.kaiyodai.ac.jp
authors.forename: Toshihide
authors.initials: T
authors.lastname: Kitakado
authors.forename: Shuichi
authors.initials: S
authors.lastname: Kitada
authors.forename: Hirohisa
authors.initials: H
authors.lastname: Kishino
authors.forename: Hans Julius
authors.initials: HJ
authors.lastname: Skaug
mesh_terms: D000465:Algorithms; D000818:Animals; D004606:Elephants / Q000235:genetics; D005399:Fishes / Q000235:genetics; D005787:Gene Frequency / Q000235:genetics*; D005828:Genetics, Population / Q000379:methods; D016013:Likelihood Functions; D008957:Models, Genetic; D009010:Monte Carlo Method