Stats Man Nate Silver Releases His First Book, The Signal and the Noise
Picture future statistician and FiveThirtyEight/New York Times blogger Nate Silver in 1984, brandishing a bat at home plate during a Little League game in East Lansing, Michigan.
His idols, the Detroit Tigers, will win the World Series that year, but six-year-old Nate isn't fantasizing about the Commissioner's Trophy—he's soberly assessing his level of talent and weighing his chances of success. He concedes he's not a good hitter but thinks he's pretty fast, so he envisions a counterintuitive strategy: "The pitchers can't pitch . . . draw a lot of walks . . . get on base and then steal." Even at this early age, in an environment where pressures from peers and parents would have demanded a simple, aspirational model for batsmanship—something like "hit a home run with the bases loaded"—Silver knows to ignore the clamor of the crowd and make a decision informed by hard truths, even if it doesn't make him look good.
"It was such a wussy style of playing," he admits now, but "it was helping my team, so it was totally logical."
Back then, he knew enough not to share such dispassionate strategies with his teammates, but since then, in the worlds of sports, business, and political analysis, Silver has made his name (first as anonymous blogger "Poblano") by applying his jaundiced eye to the world of forecasting. He began with a brief career in professional poker, followed by the development of PECOTA, a system for predicting the performance of Major League Baseball players, and crowned his achievements with the blog FiveThirtyEight.
That site proved to be one of the most accurate political meta-polls during the 2008 presidential race (he called every state except Indiana) and continues to testify to Silver's influence. So swift was his rise to King of Geekdom (Time named him to its 100 Most Influential People list in '09) that his followers lacked a bible. Now, though, he's releasing his first book, The Signal and the Noise (The Penguin Press, 352 pp., $27.95), a substantial, wide-ranging, and potentially important gauntlet of probabilistic thinking based on actual data thrown at the feet of a culture determined to sweep away silly liberal notions like "facts." For Silver, the key to successful prognostication is a clear-eyed examination of the difference between "noise," misleading or biased methods or faulty data sets, and "signal"—that which is likely to turn out to be true, and whose significance often seems obvious in hindsight.
In 13 chapters, he covers a panorama of the unpredictable and the state of mankind's ability to conquer it. Or come close, anyway. He analyzes our failure to predict the implosion of the housing bubble; he compares our talent for predicting the paths of hurricanes with our ineptitude at auguring massive earthquakes. His PECOTA radar picks up Dustin Pedroia before he becomes an MVP. He spills the secrets of TV meteorologists, debunks TV pundits (The McLaughlin Group is "explicitly intended as slapstick entertainment for political junkies"), and ponders the likelihood of a SARS epidemic. He profiles successful sports gambler Bob Voulgaris, separates the data from the hysteria surrounding global warming, and tells us how afraid of terrorism we should be (about as much as we should be scared of earthquakes).
Despite its astoundingly broad range of topics, The Signal and the Noise isn't a haphazard compilation of magazine articles; it feels as if Silver wrote it in one long toss of his Merlin's cape. He ties his baker's dozen of predictions together through the work of 18th-century English minister and statistician Thomas Bayes, a somewhat shadowy figure whose posthumous "An Essay Towards Solving a Problem in the Doctrine of Chances," writes Silver, "is a statement . . . about how we learn about the universe: that we learn about it through approximation, getting closer and closer to the truth as we gather more evidence." In the book, he cites early Bayesian Richard Price's analogy: "A person emerges into the world and sees the sun rise for the first time. At first, he does not know whether this is typical or some kind of freak occurrence. . . . Gradually . . . the probability he assigns to his prediction that the sun will rise again tomorrow approaches (though never exactly reaches) 100 percent. This contrasted with the more skeptical viewpoint of . . . David Hume, who argued that since we could not be certain that the sun would rise again, a prediction that it would was inherently no more rational than one that it wouldn't."
Bayes's theorem seems as self-evident as a Malcolm Gladwell thesis, but it has grand implications for a society so much in the thrall of spin, fantasy, overconfidence, and hope. "This is not a postmodern kind of book," says Silver with a nervous chuckle over his turkey, bacon, and cheese club sandwich at the Clinton Park Café in Brooklyn. "It's saying there is truth, but we can't know it, and it's hard for people to accept both those propositions. It requires you to accept that you'll always be a flawed, imperfect creature who's struggling to get better. Bayesianism recognizes the imperfection of our knowledge . . . so you have a starting point where we say, 'Here are our assumptions, and here are our beliefs,' and then you set up a structure to evaluate new information with those beliefs and biases."
As far as Silver is concerned, no one in a business or institution that relies on prediction can afford to accept their preconceived notions as fact. "There are different biases in all of these fields," he says, sometimes even within the same field. The implications can seem relatively inconsequential, as in the "wet bias" of meteorologists, who customarily overestimate the chance of rain because they won't lose face if they're incorrect. Of course, in certain situations, acting on the basis of forecasting can become a life-and-death issue, as in 2005, when 80,000 people in New Orleans did not evacuate despite data from the Hurricane Center that turned out to be relatively accurate, and 1,600 of them lost their lives as a result of this, as well as various logistical glitches.
Likewise, in the case of financial disasters like the crash of 2008, which Silver addresses in the first chapter, "A Catastrophic Failure of Prediction," the industry's failure to acknowledge that the housing bubble would someday burst reflected an unwillingness in certain crucial areas to pay attention to unwelcome data sets. Silver argues in the book that though "the housing bubble was discussed about 10 times per day in reputable newspapers and periodicals, the possibility of a housing bubble, and that it might burst . . . represented a threat to the ratings agencies' gravy train," and so they failed to downgrade the default rates on AAA-rated CDOs, hence the gigantic crash that nearly toppled the U.S. economy.
As an openly gay man and an Obama supporter (though he's not openly a Democrat), Silver's biases might seem obvious, but his greater love of pragmatism and number-crunching sometimes create false impressions themselves. "In 2010, when we had Republicans winning a lot of seats—correctly—people were asking, 'Why has Nate become a Republican?' Literally," he says. He sounds a bit wistful about the stir created by 2008's losing team—"Whatever you say about John McCain and Sarah Palin politically, they were interesting people"—and typically cautious about the 2012 presidential election, because people often mistake probability for prophecy.
"I'm a lot more worried about being wrong this year," he says, "because four years ago, there was nothing to lose. I know that even if I say Obama's a 75 percent to 25 percent favorite . . . that means there's a chance that Romney's going to win, and if that one in four occurs, I'll get endless shit for that."
Get the ICYMI: Today's Top Stories Newsletter
Catch up on the day's news and stay informed with our daily digest of the most popular news, music, food and arts stories in New York, delivered to your inbox.