title: Deviation of Zipf's and Heaps' Laws in human languages with limited dictionary sizes.
abstract: Zipf's law on word frequency and Heaps' law on the growth of distinct words are observed in Indo-European language family, but it does not hold for languages like Chinese, Japanese and Korean. These languages consist of characters, and are of very limited dictionary sizes. Extensive experiments show that: (i) The character frequency distribution follows a power law with exponent close to one, at which the corresponding Zipf's exponent diverges. Indeed, the character frequency decays exponentially in the Zipf's plot. (ii) The number of distinct characters grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form, and eventually saturates. A theoretical model for writing process is proposed, which embodies the rich-get-richer mechanism and the effects of limited dictionary size. Experiments, simulations and analytical solutions agree well with each other. This work refines the understanding about Zipf's and Heaps' laws in human language systems.
authors: Institute of Information Economy, Hangzhou Normal University , Hangzhou 310036, People's Republic of China.
authors: Linyuan
authors: L
authors: Lü
authors: Zi-Ke
authors: ZK
authors: Zhang
authors: Tao
authors: T
authors: Zhou
references: Nature. 2003 Aug 21;424(6951):900
references: 12931177
references: Proc Biol Sci. 2001 Nov 7;268(1482):2261-5
references: 11674874
references: Nature. 2007 Oct 11;449(7163):713-6
references: 17928859
references: Science. 2002 Nov 22;298(5598):1569-79
references: 12446899
references: Proc Natl Acad Sci U S A. 1999 Jul 6;96(14):8028-33
references: 10393942
references: Phys Rev E Stat Nonlin Soft Matter Phys. 2001 Feb;63(2 Pt 2):025101
references: 11308527
references: Phys Rev A Gen Phys. 1988 Jul 1;38(1):364-374
references: 9900174
references: Proc Natl Acad Sci U S A. 2003 Feb 4;100(3):788-91
references: 12540826
references: Proc Biol Sci. 2001 Dec 22;268(1485):2603-6
references: 11749717
references: Sci Rep. 2012;2:313
references: 22423321
references: Science. 1999 Oct 15;286(5439):509-12
references: 10521342
references: PLoS One. 2009 Nov 11;4(11):e7678
references: 19907645
references: PLoS One. 2010 Dec 02;5(12):e14139
references: 21152034
references: Proc Natl Acad Sci U S A. 2005 Nov 1;102(44):15747-51
references: 16236730
references: J R Soc Interface. 2012 Aug 7;9(73):1956-64
references: 22337632
references: Proc Natl Acad Sci U S A. 2009 Jun 30;106(26):10511-5
references: 19506244
references: Proc Natl Acad Sci U S A. 2007 Jan 30;104(5):1461-4
references: 17244704
references: Nature. 2005 May 12;435(7039):207-11
references: 15889093
references: Phys Rev Lett. 1995 May 29;74(22):4559-4562
references: 10058537
references: PLoS One. 2009;4(4):e5372
references: 19401762
references: Nature. 2002 Jun 6;417(6889):611-7
references: 12050656