title: Zipf's law holds for phrases, not words.
abstract: With Zipf's law being originally and most famously observed for word frequency, it is surprisingly limited in its applicability to human language, holding over no more than three to four orders of magnitude before hitting a clear break in scaling. Here, building on the simple observation that phrases of one or more words comprise the most coherent units of meaning in language, we show empirically that Zipf's law for phrases extends over as many as nine orders of rank magnitude. In doing so, we develop a principled and scalable statistical mechanical method of random text partitioning, which opens up a rich frontier of rigorous text analysis via a rank ordering of mixed length phrases.
authors.affiliation: Department of Mathematics &Statistics, Vermont Complex Systems Center, Computational Story Lab, &the Vermont Advanced Computing Core, The University of Vermont, Burlington, VT 05401.
authors.forename: Jake
authors.initials: J
authors.lastname: Ryland Williams
authors.affiliation: Department of Mathematics, University of Colorado, Boulder CO, 80309.
authors.forename: Paul R
authors.initials: PR
authors.lastname: Lessard
authors.affiliation: Center for Computational Engineering, Massachusetts Institute of Technology, Cambridge, MA, 02139.
authors.forename: Suma
authors.initials: S
authors.lastname: Desu
authors.affiliation: Department of Mathematics &Statistics, Vermont Complex Systems Center, Computational Story Lab, &the Vermont Advanced Computing Core, The University of Vermont, Burlington, VT 05401.
authors.forename: Eric M
authors.initials: EM
authors.lastname: Clark
authors.affiliation: Department of Mathematics &Statistics, Vermont Complex Systems Center, Computational Story Lab, &the Vermont Advanced Computing Core, The University of Vermont, Burlington, VT 05401.
authors.forename: James P
authors.initials: JP
authors.lastname: Bagrow
authors.affiliation: Department of Mathematics &Statistics, Vermont Complex Systems Center, Computational Story Lab, &the Vermont Advanced Computing Core, The University of Vermont, Burlington, VT 05401.
authors.forename: Christopher M
authors.initials: CM
authors.lastname: Danforth
authors.affiliation: Department of Mathematics &Statistics, Vermont Complex Systems Center, Computational Story Lab, &the Vermont Advanced Computing Core, The University of Vermont, Burlington, VT 05401.
authors.forename: Peter
authors.initials: P
authors.lastname: Sheridan Dodds
mesh_terms: D057225:Data Mining / Q000379:methods*; D006801:Humans; D007802:Language; D008962:Models, Theoretical