The Missing Failures: Survivorship Bias
What you can see may be only what survived, and judging from that alone leads to wrong answers. With made-up planes and 1,024 coin-flip forecasters, check the side that disappeared.
Black dots show hits on the 56 planes that came back. Pick 2 more places for armor.
● marks on planes that returned ✕ marks on planes that did not ▣ armor (hatched) ×n = marks
How to play
The two buttons at the top switch between two activities. In "Where to armor?", look at the bullet marks on the made-up planes that came back and pick two places to add armor. Then press "Show the lost planes". Your goal is to save as many planes as possible. In "The success secret", each press of "Next flip" moves the coin-flip guessing along by one. People who guess right 10 times in a row are the "winners". In both, "Auto" runs to the end and shows the explanation. Hints come in order: a way to think, a clue, then the answer.
Learn more
Every plane, number and person here is made up. This is a simplified model for learning, not a reconstruction of real events.
We often see only what made it through a filter, and the things that did not make it are out of sight. Conclusions drawn from that are skewed. This is called survivorship bias, a kind of selection bias in statistics.
The plane story is tied to Abraham Wald, who worked in the Statistical Research Group at Columbia University in the United States during the Second World War and studied how to judge a plane's weak spots from the damage on planes that returned. The group put out its memoranda in 1943, and they were reprinted in 1980. The tidy version, "armor the places with few marks", is told very often, but I could not confirm that his own memos say it that way. So it is presented here only as a story that is often told, and the plane drawing and every number are my own inventions.
The coin-flip side can be checked with arithmetic. The chance of calling 10 flips in a row is (1/2)^10, so on average the number of people out of 1,024 who do it is 1024 × (1/2)^10 = 1. That one person may be there by luck alone, yet with a few "tips" they look like a real winner. To study the habits of winners you must also study the people who did the same thing and vanished, or you will mistake cause and effect.
Money has a similar case. Poorly performing investment funds get closed or merged and drop out of the records, so averaging only the funds that remain makes the results look better than they were (1990s studies include Brown and co-authors, and Elton and co-authors). The size of the effect is not quoted here.
Rather than scold survivorship bias as a mistake, treat it as a cue: go and look for the data that went missing.