We only see the survivors

“The story of a successful person”, “a building that has stood for ages”, “a plane that made it home”. In each case, what is in front of us is only what remained. What disappeared is invisible. If we draw conclusions from the visible part alone, the answer comes out skewed. This is called survivorship bias, one kind of selection bias (a conclusion is skewed when the sample is chosen in a skewed way). The site’s “Survivorship Bias” is a simplified educational model, not a detailed reproduction of real statistics or history.

The story of the planes that came back

During the Second World War, the mathematician Abraham Wald, at the Statistical Research Group at Columbia University in the United States, is said to have worked on estimating how vulnerable each part of an aircraft was from the damage on aircraft that returned. The work was classified and not published at the time. In 1980 it was reprinted under the title “A Method of Estimating Plane Vulnerability Based on Damage of Survivors”. As the title says, the method takes into account that only the planes that came back can be seen.

The story is often told like this: “The returning planes had many holes in the wings and body, and few around the engine. The military wanted to protect the places with many holes, but Wald said the opposite: protect the places with no holes, because planes hit there did not come back.” This is an often-told story, however, and I could not confirm that Wald’s own documents said so. The core idea is sound, but the details read like legend.

Let us take just the idea. Suppose shots hit anywhere on a plane equally. But some parts bring a plane down when hit, while others let it return anyway. If you then count only planes that returned, the “dangerous parts” show few holes, because the planes hit there are not among the ones you can count. In this site’s model, each section has a chance of bringing the plane down when hit, and the holes on all planes are shown beside the holes on returned planes. The share of holes among returned planes is smaller the higher a section’s chance of bringing the plane down.

Funds that vanish

The same thing happens with investment funds, which manage money. Poorly performing funds get closed or merged into others and disappear from the records. If you average only the funds that remain, the results look better than they really were. Researchers addressed this problem in papers from the 1990s. This article does not go into detailed numbers.

The person on a winning streak

Say 1,024 people each flip a coin every turn for a year, winning or losing by the result. By chance alone, on average one person will call it right ten times in a row (1,024 × 1/2 to the 10th power = 1). Looking only at that person and saying “talented” is survivorship bias. The misses of the other 1,023 people have become invisible. This site’s program checked the calculation “among n people, the average number with a streak of k is n ÷ 2 to the k”.

“Advice from successful people” works the same way. We hear little from the many who did the same things and failed. Looking only at the successful, we cannot tell whether the habit or luck explained their success.

Were old buildings sturdier?

People sometimes say old buildings were sturdier. One explanation is that poor buildings were demolished first, and only the remaining ones catch our eye. There are objections to that explanation, and the matter is not settled. It may not be the whole reason. On this site the building model is only an example of how the mechanism works; it does not say that real buildings behave this way.

What should we watch for?

Survivorship bias is not something to blame as a “statistical mistake”. It is a point of view. Asking “what is on the side I cannot see?” can change a conclusion. Look for data from the side that failed or disappeared.

Spot the differences

• Everything versus what remained: counting everything and counting only what remained give different distributions.
• Chance versus talent: looking only at the streak winners hides the power of luck.
• Legend versus record: treat a widely told story and a confirmed record separately.

Switch between showing everything and showing only the survivors, and see what disappears.