Suppose you brought all the smartest baseball kids on the block together, locked them in a house for two weeks with an unlimited supply of Twinkies and beer, and asked them to pool all their statistical tools to predict the upcoming baseball season? What might the result look like?
SBNation’s Jason Brannon did that – well, he didn’t actually lock them up, but he combined (or averaged) several of the most egg-headed systems from the 2011 season. and compared to the actual results. The predictors he used were PECOTA (developed by Nate Silver for Baseball Prospectus), Marcel (developed by Jack Sackman and Tom Tango and named for a monkey named Marcel), CAIRO (an advanced version of Marcel) and BP-cofounder Clay Davenport’s DTs (for Davenport Translations, which are so complicated you’ll have to Google him and figure it out yourself).
Here’s a link to Brannon’s piece, but here’s a short form look at the Yankees’ and Mets’ divisions:
Team, projected 2011 wins, actual 2011 wins, difference
Boston 95, 90 -5
New York 92, 97 +5
Tampa Bay 87, 91 +4
Baltimore 79, 69 -10
Toronto 74, 81 +7
Philadelphia 90, 102 +12
Atlanta 87, 89 +2
Florida 82, 72 -10
New York 80, 77 -3
Washington 73, 80 +7
What would you consider an accurate prediction? Myself, I’d say you
have to get it within 2-3 games to have a claim that you had it right.
By that standard, the projection for 87 wins for the Braves last year
and their actual total of 89 was spot-on. The projection that the Mets
would win 80 when they actually finished with 77 also was pretty good.
But the experts didn’t project a single exact win total for any team;
the closest was the one-game difference between the Pirates, projected
to win 71 and coming up with 72, and only three teams – the Braves,
Dodgers and Giants — within two.
The worst predictions were Minnesota – winning 63 games against the
projected 84, though we must cut the prognosticators some slack as the
Twins’ two superstars, Joe Mauer and Justin Morneau, were out for most
of the season – and Arizona , 73 actual vs. 94 projected, which I would
file under the heading of “Who cares”?
I’d say this proves one thing about any system based on sabermetrics:
they do a much better job predicting the past than the future.