title: Understanding Zipf's law of word frequencies through sample-space collapse in sentence formation.
abstract: The formation of sentences is a highly structured and history-dependent process. The probability of using a specific word in a sentence strongly depends on the 'history' of word usage earlier in that sentence. We study a simple history-dependent model of text generation assuming that the sample-space of word usage reduces along sentence formation, on average. We first show that the model explains the approximate Zipf law found in word frequencies as a direct consequence of sample-space reduction. We then empirically quantify the amount of sample-space reduction in the sentences of 10 famous English books, by analysis of corresponding word-transition tables that capture which words can follow any given word in a text. We find a highly nested structure in these transition tables and show that this 'nestedness' is tightly related to the power law exponents of the observed word frequency distributions. With the proposed model, it is possible to understand that the nestedness of a text can be the origin of the actual scaling exponent and that deviations from the exact Zipf law can be understood by variations of the degree of nestedness on a book-by-book basis. On a theoretical level, we are able to show that in the case of weak nesting, Zipf's law breaks down in a fast transition. Unlike previous attempts to understand Zipf's law in language the sample-space reducing model is not based on assumptions of multiplicative, preferential or self-organized critical mechanisms behind language formation, but simply uses the empirically quantifiable parameter 'nestedness' to understand the statistics of word frequencies.
authors.affiliation: Section for Science of Complex Systems, CeMSIIS, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA IIASA, Schlossplatz 1, 2361 Laxenburg, Austria stefan.thurner@meduniwien.ac.at.
authors.forename: Stefan
authors.initials: S
authors.lastname: Thurner
authors.affiliation: Section for Science of Complex Systems, CeMSIIS, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria.
authors.forename: Rudolf
authors.initials: R
authors.lastname: Hanel
authors.affiliation: Section for Science of Complex Systems, CeMSIIS, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria.
authors.forename: Bo
authors.initials: B
authors.lastname: Liu
authors.affiliation: Section for Science of Complex Systems, CeMSIIS, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria.
authors.forename: Bernat
authors.initials: B
authors.lastname: Corominas-Murtra
mesh_terms: D001877:Books; D006801:Humans; D007802:Language; D008962:Models, Theoretical
keywords: language formation; random walks on networks; scaling in stochastic processes; word-transition networks