#zipf's law

[linguisticalities]

Almost All Languages Appear To Follow Zipf's Law, And We Have No Idea Why

Even the languages we haven't deciphered yet, such as the one in the Voynich Manuscript, appear to follow this rule.

[phonetish]

Unzipping Zipf's Law: Solution to a century-old linguistic problem

Did you know that in every language, the most frequent word occurs twice as often as the second most frequent word? This phenomenon called 'Zipf's law' is more than one century old, but until now, scientists have not been able to elucidate it exactly. Sander Lestrade, a linguist at Radboud University in The Netherlands, proposes a new solution to this notorious problem in PLOS ONE.

Did you know that in every language, the most frequent word occurs twice as often as the second most frequent word? This phenomenon called 'Zipf's law' is more than one century old, but until now, scientists have not been able to elucidate it exactly. Sander Lestrade, a linguist at Radboud University in The Netherlands, proposes a new solution to this notorious problem in PLOS ONE.                                

Zipf's law describes how the frequency of a word in natural language, is dependent on its rank in the frequency table. So the most frequent word occurs twice as often as the second most frequent word, three times as often as the subsequent word, and so on until the least frequent word (see Figure 1). The law is named after the American linguist George Kingsley Zipf, who was the first who tried to explain it around 1935.

Biggest mystery in computational linguistics

"I think it's safe to say that Zipf's law is the biggest mystery in computational linguistics," says Sander Lestrade, linguist at Radboud University in Nijmegen, the Netherlands. "In spite of decades of theorizing, its origins remain elusive." Lestrade now shows that Zipf's law can be explained by the interaction between the structure of sentences (syntax) and the meaning of words (semantics) in a text. Using computer simulations, he was able to show that neither syntax or semantics suffices to induce a Zipfian distribution on its own, but that syntax and semantics 'need' each other for that.

[factoidfactory]

Random Fact #906

Zipf’s Law states that while few items occur often, many items occur rarely.

For example, in an early citation study in 1927 researchers found that very few journals were cited frequently in the Journal of the American Chemical Society, while many journals were only cited once.

[cybernetic-world]

[thekingjameshrmh]

The of and to a in is I that it for you was with on as have but be they.

Of the hundred most frequently used words in English, ninety-six have Germanic roots.[1]

The frequency of a word is expected to be approximately one over its popularity index and, conforms to Zipf's law.[2]

KING JAMES HRMH

rel: [1] ELT Global Blog. "Ten things you might not have known about the English language" blog.oxforddictionaries.com, Aug. 12, 2015, blog.oxforddictionaries.com/2015/08/12/ten-things-you-might-not-have-known-about-the-english-language/ [2] Michael. "The Zipf Mystery" youtube.com, Sep. 15, 2015, www.youtube.com/watch?v=fCn8zs912OE

[bastionofbibliophiles]

[beatusmusicus]

The of and to a in is I that it for you was with on as have but be they... The Zipf Mystery

[plumburn]

I JUST REALIZED THAT ZIPF’S LAW APPLIES TO LITERALLY EVERYTHING

So, Zipf’s law goes a little something like this: 

“In the English language, the probability of encountering the ‘r’th most common word is given roughly by ‘P (r) = 0.1 for r‘ up to 1000 or so. The law breaks down for less frequent words, since the harmonic series diverges. Goetz states the law as follows: The frequency of a word is inversely proportional to its statistical rank r such that ‘P (r) ≈ 1/(r ln (1.78 R))’  where R is the number of different words.”   -Wolfram Mathworld

In layman’s terms: the more frequently used words get use a lot more often than the less used ones.  Vsauce one has an amazing video about this, but just in case you don’t have time, here’s the summary:  the top 5 words are 50% of what we say and 80% of what we say is the top 20% of all words.

It’s also worth pointing out that half my dash is the same 5 posts and 80% of my dash accounts for only 20% of the unique Tumblr posts on my dash.  You’re welcome.