title: The empirical Bayes estimators of fine-scale population structure in high gene flow species.
abstract: An empirical Bayes (EB) pairwise FSTestimator was previously introduced and evaluated for its performance by numerical simulation. In this study, we conducted coalescent simulations and generated genetic population structure mechanistically, and compared the performance of the EBFSTwith Nei's GST, Nei and Chesser's bias-corrected GST(GST_NC), Weir and Cockerham's θ (θWC) and θ with finite sample correction (θWC_F). We also introduced EB estimators for Hedrick' G'STand Jost' D. We applied these estimators to publicly available SNP genotypes of Atlantic herring. We also examined the power to detect the environmental factors causing the population structure. Our coalescent simulations revealed that the finite sample correction of θWCis necessary to assess population structure using pairwise FSTvalues. For microsatellite markers, EBFSTperformed the best among the present estimators regarding both bias and precision under high gene flow scenarios (FST≤0.032). For 300 SNPs, EBFSThad the highest precision in all cases, but the bias was negative and greater than those for GST_NCand θWC_Fin all cases. GST_NCand θWC_Fperformed very similarly at all levels of FST. As the number of loci increased up to 10 000, the precision of GST_NCand θWC_Fbecame slightly better than for EBFSTfor cases with FST≥0.004, even though the size of the bias remained constant. The EB estimators described the fine-scale population structure of the herring and revealed that ~56% of the genetic differentiation was caused by sea surface temperature and salinity. The R package finepop for implementing all estimators used here is available on CRAN.
authors.affiliation: Graduate School of Marine Science and Technology, Tokyo University of Marine Science and Technology, 4-5-7 Konan, Minato-ku, Tokyo, 108-8477, Japan.
authors.forename: Shuichi
authors.identifier: http://orcid.org/0000-0001-5838-0374
authors.initials: S
authors.lastname: Kitada
authors.affiliation: Graduate School of Marine Science and Technology, Tokyo University of Marine Science and Technology, 4-5-7 Konan, Minato-ku, Tokyo, 108-8477, Japan.
authors.forename: Reiichiro
authors.initials: R
authors.lastname: Nakamichi
authors.affiliation: Graduate School of Agriculture and Life Sciences, The University of Tokyo, 1-1-1 Yayoi, Bunkyo-ku, Tokyo, 113-8657, Japan.
authors.forename: Hirohisa
authors.initials: H
authors.lastname: Kishino
mesh_terms: D000818:Animals; D056808:Biostatistics / Q000379:methods*; D003198:Computer Simulation; D005399:Fishes / Q000145:classification* / Q000235:genetics*; D051456:Gene Flow; D005828:Genetics, Population / Q000379:methods*; D060005:Genotyping Techniques / Q000379:methods*; D018895:Microsatellite Repeats; D008957:Models, Genetic; D020641:Polymorphism, Single Nucleotide
keywords: Atlantic herring; SNP; empirical Bayes; microsatellite; pairwise FST