When you read interviews with successful people, you often come across stories like “I dropped out of college and started a company” or “I bet everything on it.” But then where are the people who made the same choices and quietly disappeared? This illusion, which arises when we draw conclusions only from the results left in front of us, is a trap statisticians have warned against for a long time. It is called “survivorship bias.”
Put simply, survivorship bias is “the error of judging the whole by observing only the subjects that survived some filtering process, without seeing those that dropped out.” Because the sample is skewed from the start, no matter how carefully you calculate, your conclusion can easily turn out more optimistic than reality or head in the wrong direction altogether.
This article lays out survivorship bias in an easy-to-follow way, from its definition and how it arises to the story of the World War II bombers, examples from daily life and investing, and the habit of looking for the data you cannot see.
The Structure of Survivorship Bias and Principles for Interpreting It
What Is Survivorship Bias?
Survivorship bias is an error that already creeps in at the stage of choosing the sample. If the group you want to observe has already passed through a filter such as an exam, a competition, or time, the data in your hands is not the whole but merely “the part that got through the filter.” Yet because those who dropped out leave no records or go unnoticed, we are tempted to explain the whole using only what remains.
- Filter: the process that separates subjects into those that survive and those that drop out.
- Survivors: the subjects that passed the filter and can be observed.
- Silent evidence: the subjects that dropped out and vanished from the record.
In statistics, this is regarded as one type of “selection bias” in the broad sense. The problem lies not in the calculation but in failing to ask what never made it into the sample.
The feeling that “old songs are better than today’s songs” can be explained by the same structure. The songs still sung decades later are the few, out of the countless songs released in their era, that stayed loved for a long time. The songs that were quickly forgotten never even make it onto the comparison list, so the past looks brighter than it really was.
The Bullet Holes in the Bombers That Came Back
The story most often cited to explain survivorship bias is the case of the World War II bombers. Abraham Wald, a mathematician working at the Statistical Research Group at Columbia University in the United States, was given the task of estimating how well aircraft could withstand enemy fire.
The commonly told version goes like this. When the returning aircraft were examined, there were many bullet holes in the wings and fuselage and relatively few around the engines. Many people suggested reinforcing the places with the most holes, but Wald thought the opposite. The engines showed few holes because the planes that were hit there never made it back in the first place.
(1) The holes you can see
The holes in the planes that came back show you “the places that can be hit and still hold up.”
(2) The holes you cannot see
The places where the planes that never returned were hit are most likely the fatal weak points.
Wald’s actual reports were far more mathematical than this simple anecdote. Starting from the premise that information on the planes that never returned was missing, he worked out, in a series of memos, methods for estimating the vulnerability of each part of the aircraft from the remaining data alone. The story may have been simplified, but its core insight has come down to us intact.
Put simply, the lesson is that what is missing from the data may be the most important information of all.
Survivorship Bias in Everyday Life and Investing
This trap is not confined to the battlefield. In investing, fund return statistics are often cited as a prime example. When funds that performed poorly and closed down or were merged into other funds drop out of the data, the average return of the remaining funds comes out higher than it really is.
The saying “old buildings were built to last” is similar. Only the buildings that held up for a long time are still standing, while those that were poorly built may already have been torn down and dropped out of the comparison. Business books that collect what successful companies have in common should also be read with care if they did not also examine the companies that used the same strategies and failed.
A similar problem arises in medical research. If you leave out the patients who quit a study partway through because of side effects or a lack of benefit and analyze only those who stayed to the end, the treatment can look more effective than it really is. That is why clinical trials place great importance on including every patient originally assigned in the analysis through to the end.
[Key Points from the Examples]
- First ask what filter the remaining subjects have passed through.
- Record the number and characteristics of the subjects that disappeared as well.
- Judge what successful cases have in common only after comparing them with failed cases.
Four Habits for Finding the Data You Cannot See
Survivorship bias is hard to avoid completely, but changing how you interpret data can greatly reduce its influence. The key is to first examine “how the data in front of you got here.” Once you know how the sample was formed, you can also guess which pieces are missing from it.
(1) Count everything at the starting point
Check how many started out at the beginning and how many remain. If you do not know the denominator, you cannot know the success rate either.
(2) Look for records of those that dropped out
Deliberately gather data on the subjects that disappeared, such as shops that closed, participants who gave up partway, and companies that were delisted.
(3) Set up a comparison group that made the same choice
The statement “I succeeded with this method” only becomes meaningful when you know how many people failed using the same method.
(4) State the scope of your conclusion
Make it clear in your wording whether the conclusion is “based on the companies that survived” or “based on all the companies that started out.”
The more glamorous the success stories, the smaller the number of failures hidden behind them appears. The more convincing the results in front of you seem, the more you need to step back and ask again how the sample was formed.
Asking “what do the ones that never came back tell us?” behind the survivors you can see is the first step toward reading data correctly.
