title: Entropy estimation in Turing's perspective.
abstract: A new nonparametric estimator of Shannon's entropy on a countable alphabet is proposed and analyzed against the well-known plug-in estimator. The proposed estimator is developed based on Turing's formula, which recovers distributional characteristics on the subset of the alphabet not covered by a size-n sample. The fundamental switch in perspective brings about substantial gain in estimation accuracy for every distribution with finite entropy. In general, a uniform variance upper bound is established for the entire class of distributions with finite entropy that decays at a rate of O(ln(n)/n) compared to O([ln(n)]2/n) for the plug-in. In a wide range of subclasses, the variance of the proposed estimator converges at a rate of O(1/n), and this rate of convergence carries over to the convergence rates in mean squared errors in many subclasses. Specifically, for any finite alphabet, the proposed estimator has a bias decaying exponentially in n. Several new bias-adjusted estimators are also discussed.
authors.affiliation: Department of Mathematics and Statistics, University of North Carolina at Charlotte, Charlotte, NC 28223, USA. zzhang@uncc.edu
authors.forename: Zhiyi
authors.initials: Z
authors.lastname: Zhang
mesh_terms: D015982:Bias; D003198:Computer Simulation; D019277:Entropy; D015233:Models, Statistical; D011336:Probability; D018401:Sample Size; D018709:Statistics, Nonparametric