The following is an excerpt from my book The Uncertainty Solution: How to Invest With Confidence in the Face of the Unknown. It’s from a section about why investment experts and economists are so bad at predicting the future. A major reason is overconfidence, which is fueled by something we all fall victim to called the “Illusion of Validity.”

THE ILLUSION OF VALIDITY

The 2001 St. Louis Rams were a juggernaut. Led by Hall of Famers Kurt Warner, Orlando Pace, Isaac Bruce, and Marshall Faulk, the Rams were stacked. Their prolific offense, known as the “Greatest Show on Turf,” was a scoring machine, and they were the first team to tally more than 500 points a year in three consecutive seasons. They only lost two games all year, and in the playoffs they blew out the Green Bay Packers, won a slugfest against the Philadelphia Eagles, and entered Super Bowl XXXVI as two touchdown favorites over the 11–5 New England Patriots.

The Patriots were underdogs, partly because their star quarterback, Drew Bledsoe, had been knocked out for the season earlier that year. Now the Patriots were led by a second-year, sixth-round draft pick named Tom Brady. While Brady had done well filling in for Bledsoe, pundits thought the Super Bowl would be a high-scoring game and that Brady and his offense wouldn’t be able to keep up.

The Rams outgained the Patriots during the game 427–267 in total yards. But the game was tied with 1:30 remaining when the Patriots got the ball for the final drive. Everyone expected the Patriots to run the ball a few times and send the game into overtime, but with no timeouts left, Brady completed five key passes and moved his team into field goal range. The Patriots then kicked the game winner as the clock expired.

I attended Super Bowl XXXVI and, as a die-hard Rams fan, I was devastated. I paid a ton of money for the tickets, drove ten hours from Saint Louis to New Orleans, and slept on the floor of a hotel room with ten other people because hotel rooms were so scarce. I had fun, but the upset cast a pall.

After the game, I was at a bar commiserating with some fellow Rams fans. I mentioned that, including the Super Bowl, I’d attended just three Rams games that year, and those were the only three they’d lost. Everyone at the table was stunned, wondering if anyone else had only attended their losing games. My record of attendance was unusual: typically, if you were a big enough fan to go to the Super Bowl, you’d have gone to more than just two regular season games. Soon, people started joking, blaming me for the loss. Then, some actually started to get mad at me. (To be fair, we were at a bar, and everyone was drinking.)

And it didn’t end there. Back in Saint Louis, some of my friends begged me not to go to any Rams games the following season. I ignored them and went to the first two. They lost. Wow. I went to a total of five games over two years and they lost every one—the only five the Rams had lost over the previous twenty-two. The cries for me to stop going grew louder. Even though I knew better, I was starting to feel a bit paranoid about my bad luck.

Of course, it’s ridiculous to think that one fan has any influence on a team winning or losing, but we’ve evolved to think this way. As discussed in Chapter 1, one of our key survival advantages is our ability to recognize patterns. Our brains are constantly looking for patterns to guide our way through the world. When we recognize one, we want to explain it. The problem is that many of the patterns we see are meaningless—like the pattern of my Rams game attendance. But our brains insist on finding coherence—a narrative when there is none.

This drive to discover patterns and explain them is a major factor in overconfidence. It’s why we think we’re better at predicting outcomes than we are. As long as the explanation we construct is coherent, we trust it. We can’t help it. Explanations give us confidence that we understand the world, and that makes us feel safe. As social psychologist David Dunning says in his article, “We Are All Confident Idiots,” “we are unbridled pattern recognizers and profligate theorizers.”