If you open a book and count its words one by one, a strange imbalance appears. A few words, such as "and," "is" and "thing," show up on every page without pause, while most words appear once or twice in the entire book and then vanish. Yet this imbalance is not haphazard; it follows a remarkably consistent ratio. The term for this rule is "Zipf's Law."
Put simply, Zipf's Law is the empirical law that "if you rank words in order of how often they are used, the number of times each word is used is inversely proportional to its rank." Together with Bradford's Law and Lotka's Law, it is counted among the three major laws of bibliometrics, and today it also underlies search engines and indexing technology.
In this article, we will look in turn at how Zipf's Law became known, what the law says and how its causes have been explained, and its uses and limitations in libraries and information retrieval.
The Order of Words Seen Through Zipf's Law
What is Zipf's Law?
The law is named after the American linguist George Kingsley Zipf. While teaching German at Harvard University in the 1930s, he counted how often words appeared in texts in various languages and set out a simple relationship between rank and frequency.
However, Zipf was not the first to discover this relationship. The French stenographer Jean-Baptiste Estoup had already recorded the same phenomenon in 1916, and Zipf himself acknowledged this. Even so, because it was Zipf who extended the rule to many languages and fields and made it widely known, it is still called by his name today.
In his 1949 book Human Behavior and the Principle of Least Effort, Zipf extended this idea to human behavior as a whole.
The rule does not stop at words. In 1913, the German physicist Felix Auerbach had already observed that when cities are ranked by population, their populations are inversely proportional to their rank, and similar patterns were later confirmed in many other kinds of data, such as company sizes and the frequency of notes in music.
The inverse relationship between rank and frequency
Zipf's Law can be expressed as a very simple ratio. If you take the count of the most frequently used word as the baseline, the count of each remaining word is close to that baseline divided by its rank.
(1) The top-ranked word
The most common word serves as the baseline. In the Brown Corpus, a collection of American English texts of about one million words, "the" appears 69,971 times, accounting for nearly 7% of the total.
(2) The second-ranked word
It is used about half as often as the top word. In the Brown Corpus, the second-ranked word, "of," appears 36,411 times, slightly more than 3.5% of the total.
(3) Words ranked third and below
The third-ranked word drops to about one-third of the top word, and the tenth to about one-tenth. Because usage is so concentrated at the top, just 135 words account for half of all word occurrences in this collection.
If you plot the graph with both the horizontal and vertical axes on logarithmic scales, this relationship appears as an almost straight line running down to the lower right.
Zipf explained this phenomenon with the principle of least effort. Speakers try to convey many meanings with a small number of words, while listeners want a variety of words with clear meanings, and the distribution arises as these two forces balance each other.
Uses in libraries and information retrieval
Zipf's Law is used to pick out which words in a text best reveal its content. Words that are too common appear in every text, and words that are too rare have often just slipped in by chance.
- Stop Word List: the most frequent words, such as particles and articles, are removed from the index in advance, saving storage space and search time.
- Keyword Extraction: in 1958, Hans Peter Luhn of IBM experimented with having computers produce abstracts of papers by cutting out words that were too common and too rare and selecting mid-frequency words as keywords.
- Authorship Attribution: because each person's pattern of word frequencies is relatively consistent, it is used as evidence for determining who wrote a text.
- Collection and Index Planning: knowing that use is concentrated on a small number of materials and terms makes it possible to set criteria for acquiring and organizing the most frequently sought items first.
Lotka's Law, under which the number of papers is concentrated among a few authors, is explained mathematically as a distribution from the same family as Zipf's Law, and Bradford's Law, under which papers are concentrated in a small number of journals, shows a similar concentration. This is why the three laws of bibliometrics are grouped together.
Points to keep in mind when reading Zipf's Law
Zipf's Law holds surprisingly often, but it is an approximation drawn from experience rather than an exact law of nature. Those who use it therefore need to keep a few limitations in mind.
Frequently noted limitations
- Deviation at the Extremes: for the highest-ranked and lowest-ranked words, actual frequencies come out quite differently from the ideal ratio.
- Language Differences: languages such as Chinese and Vietnamese, in which a single syllable forms a unit of meaning, sometimes deviate considerably from the law.
- Debate over Causes: similar distributions appear even in nonsense text produced by typing letters at random, which has led to the criticism that the principle of least effort alone is not a sufficient explanation.
In fact, in a study that analyzed 30,000 English texts, the share of texts that fit the law well ranged from about 15% to 40% depending on how the law was defined. Using the cumulative distribution, it was about 40%, but that was still less than half. The law is only a map that points the way, not the right answer for every text.
Zipf's Law shows with a simple ratio that the language we use relies heavily on a small number of words. This rule, obtained about 90 years ago by counting words by hand, still lives on throughout information technologies such as stop word handling and keyword extraction.
Picking out the truly meaningful words from between the common and the rare is the first step in information retrieval.
