title: Rediscovery of Good-Turing estimators via Bayesian nonparametrics.
abstract: The problem of estimating discovery probabilities originated in the context of statistical ecology, and in recent years it has become popular due to its frequent appearance in challenging applications arising in genetics, bioinformatics, linguistics, designs of experiments, machine learning, etc. A full range of statistical approaches, parametric and nonparametric as well as frequentist and Bayesian, has been proposed for estimating discovery probabilities. In this article, we investigate the relationships between the celebrated Good-Turing approach, which is a frequentist nonparametric approach developed in the 1940s, and a Bayesian nonparametric approach recently introduced in the literature. Specifically, under the assumption of a two parameter Poisson-Dirichlet prior, we show that Bayesian nonparametric estimators of discovery probabilities are asymptotically equivalent, for a large sample size, to suitably smoothed Good-Turing estimators. As a by-product of this result, we introduce and investigate a methodology for deriving exact and asymptotic credible intervals to be associated with the Bayesian nonparametric estimators of discovery probabilities. The proposed methodology is illustrated through a comprehensive simulation study and the analysis of Expressed Sequence Tags data generated by sequencing a benchmark complementary DNA library.
authors.affiliation: Department of Economics and Statistics, University of Torino and Collegio Carlo Alberto, Torino, Italy.
authors.forename: Stefano
authors.initials: S
authors.lastname: Favaro
authors.affiliation: Department of Economics and Statistics, University of Torino and Collegio Carlo Alberto, Torino, Italy.
authors.forename: Bernardo
authors.initials: B
authors.lastname: Nipoti
authors.affiliation: Department of Statistics, University of Oxford, Oxford, United Kingdom.
authors.forename: Yee Whye
authors.initials: YW
authors.lastname: Teh
mesh_terms: D000465:Algorithms; D001499:Bayes Theorem; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D020224:Expressed Sequence Tags; D000069550:Machine Learning; D015233:Models, Statistical; D010363:Pattern Recognition, Automated / Q000379:methods*; D017422:Sequence Analysis, DNA / Q000379:methods*
keywords: Asymptotic equivalence; Bayesian nonparametrics; Credible intervals; Discovery probability; Expressed Sequence Tags; Good-Toulmin estimator; Good-Turing estimator; Smoothing technique; Two parameter Poisson-Dirichlet prior