Solvable null model for the distribution of word frequencies.
Zipf's law asserts that in all natural languages the frequency of a word is inversely proportional to its rank. The significance, if any, of this result for language remains a mystery. Here we examine a null hypothesis for the distribution of word frequencies, a so-called discourse-triggered word choice model, which is based on the assumption that the more a word is used, the more likely it is to be used again. We argue that this model is equivalent to the neutral infinite-alleles model of population genetics and so the degeneracy of the different words composing a sample of text is given by the celebrated Ewens sampling formula [Theor. Pop. Biol. 3, 87 (1972)]], which we show to produce an exponential distribution of word frequencies.
Fontanari
J F
JF
Instituto de Física de São Carlos, Universidade de São Paulo, Caixa Postal 369, 13560-970 São Carlos, São Paulo, Brazil.
Perlovsky
L I
LI
D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D015233:Models, Statistical; D009323:Natural Language Processing; D012660:Semantics; D016008:Statistical Distributions; D009626:Terminology as Topic; D018875:Vocabulary, Controlled