Much has been made of StarsDraft’s aggressive data driven approach to player pricing. StarsDraft salaries have been touted as the most predictive salaries in the industry – a claim that people love to challenge week in and week out. But what does the data say? Are the salaries really all that predictive after all?
In order to get a handle on how predictive our salaries were I created a script to compare our salaries against industry experts and identify where we were right and where we were wrong. There are a few important caveats to cover before we dive into the analysis. In this article I’m comparing against the projections of CBS fantasy expert Jamey Eisenburg. Jamey co-hosts an excellent podcast, Fantasy Football Today – if you haven’t checked it out I highly recommend it. I’ve listened to Jamey and his co-hosts Dave Richard and Adam Aizer for a few years now as part of my weekly fantasy due diligence – it’s consistently both entertaining and educational. I’ve converted Jamey’s projections into the StarsDraft scoring format to make sure they are readily comparable.
I chose to compare against Mr. Eisenburg for several reasons. First, Jamey is historically the most accurate of the CBS crew. Also Jamey (and much to their credit all the analysts at CBS) is one of the few experts that publish detailed numerical projections to accompany and support his player rankings. CBS also makes their historical projections readily available for prior weeks on CBS.com. The majority of free fantasy content providers either focus only on rankings without any numerical projections (eg. Yahoo! Sports), or they focus only on the coming week without making their historical performance easily accessible from prior weeks (eg. ESPN). It’s exceptionally hard to find experts who both publish the low-level stat predictions that factor into their final player projections/ranks and keep a historical record of their projections for validation after the fact. CBS does a great job on both fronts, and this makes Mr. Eisenburg’s projections a prime candidate to begin our comparison.
One more important note on this analysis: StarsDraft.com salaries are published along with our contests on Monday afternoon of each week. That means the projections on which the salaries are based were made Monday morning and cannot change throughout the week. Most analysts, including that of Mr. Eisenburg and the CBS crew, get the added benefit of an entire week of injury updates, breaking news, or Twitter buzz plus additional time to bounce ideas off of other experts in the industry in order to arrive at their final Sunday morning projections. Obviously this provides a distinct informational edge to the experts – the same type of edge that underpins much of a user’s daily fantasy sports skill.
Alright, without further ado – let’s look at the data.
Perhaps the most straightforward way to compare fantasy football predictions is to compare the absolute rankings of two rankers against the actual ranks of each player for the week. Treating both StarsDraft’s projections and those from CBS as ranks, we see the following (restricted to athletes who scored 1 or more fantasy points in the week):
In summary: of the 235 players analyzed, StarsDraft.com ranks outperformed CBS 59.4% of the time. StarsDraft ranks were superior at every position, with the smallest edge at the RB position where StarsDraft’s rank was closer to actual only 52.1% of the time. The QB, WR, and TE positions were ranked closer to actual 61.8%, 62.1% and 65% of the time, respectively.
Of course rankings are only half the story. Sure StarsDraft.com nailed Le’veon Bell as the number 1 ranked RB going into week 14 – but we didn’t project him to outscore the number 2 RB by a whopping 11 points. Below is a comparison of the actual projections, including a measurement of the average size of the error by position and overall:
When comparing the raw projections CBS fares a bit better, with StarsDraft.com projections outperforming CBS only 56.6% of the time. By position StarsDraft.com out-projected CBS on 55.9% of QBs, 53.4% of RBs, 54.4% of WRs and a whopping 70% of TEs and at every position StarsDraft.com had a lower average error.
One thing that became evident when comparing these two sets of predictions is that humans are much more easily star-struck than machines. Week 14 happened to be one of the highest fantasy scoring weeks of the season and we see the Mr. Eisenburg more accurately predicting almost all of the top-scoring players, but missing much more frequently in the mid-range or low-end of actual performers on account of this overestimation. In fact, CBS’s projections for the top 10 RBs (excluding Andre Williams) and the top 6 WRs were closer to their actual performance. Systematic error, however, should not be mistaken for foresight. It’d be easy to accurately project the top 10 WRs if you simply projected every WR at 20-25 points. On average you’d be very, very wrong – but for whoever ended up in the top 10 WR spots you’d probably fare better than an algorithm that attempted to give a reasonable best guess for each player. For every Jordy Nelson and A.J. Green that is successfully projected at over 20 points (and end up putting up 30 points or more), there is a Dez Bryant, Emmanuel Sanders, Deandre Hopkins or Demaryius Thomas that underperformed their lofty CBS projections by 9 points or more (to be sure, they also underperformed their StarsDraft projections by 5 or more).
This bias by the experts towards projecting studs systematically higher than what a sensible data-driven model would suggest is endemic and totally understandable. Expert picks are highly scrutinized and there is a strong incentive not to deviate too far from the consensus view. As it happens the consensus view appears to be strongly influenced by reporting bias and selective reporting . Think back on the fantasy media you’ve consumed pertaining to week 14. Which was more heavily discussed, Demaryius Thomas scoring 2.1 points, or Julio Jones scoring 37? Both are equally exceptional and both have serious impacts for each player’s fantasy prospects moving forward. Yet the fantasy community tends to discard negative performances as totally inconsequential and take a lazer-like focus on every upside surprise. To put this effect in perspective for this particular case, Demaryius has only scored more than 20 fantasy points 4 times in 13 games this year and has scored less than 10 in 3 and less than 3 in 1 (last week). I imagine that he was projected at over 20 points going into this week by most industry experts for a .5 PPR format. If an expert predicts Demaryius at this lofty level and he puts up only 2.1 points no one is going to question the call… I mean, c’mon, it’s Demaryius freakin’ Thomas! If, on the other hand, the expert projects Thomas at a more realistic number (say, our algorithmic projection of 18.2) and he goes off they are likely to be called on the carpet for being too bearish on an obvious stud. The fact that their projection really was the long-term most accurate possible estimate of player performance for that week not withstanding. Given that player performance is highly variable, and under-projections are much more visible than are over-projections, there is a non-trivial incentive for experts to overstate the expectations of star players.
The most commonly heard refrains from DFS users familiar with other sites or users converted from season-long formats when they first encounter StarsDraft.com salaries is that TEs are obviously underpriced. It seems that even experts aren’t immune to the bias of thinking TEs are more consistent, productive, and reliable than they actually are. The fact that StarsDraft.com salaries were 65% better at ranking TEs and a whopping 70% better at projecting actual TE output is an indication that the thinking about TEs across the industry just doesn’t jive with the numbers.
In my years as a data scientist I’ve encountered a wide variety of complex problems, most of them involving making predictions in the face of uncertainty. Predicting fantasy sports outcomes is definitely in the top 5 in terms of challenge and complexity.
Week in and week out there is so much variance; coaching strategies are always changing, mid-week injuries can’t be avoided, and inevitably some insane weather system will threaten what would otherwise have been a fantasy bonanza (to use a CBS podcast coined turn of phrase). All of these factors conspire against any analyst trying to project athlete performance; much more so those trying to make projections a full week in advance. All things considered it would not have shocked me at all if the expert’s up-to-the-minute Sunday morning projections managed to out-project and out-rank StarsDraft’s week old salaries … But they didn’t. At least not in week 14.
I can’t wait to continue this analysis moving forward and I’m certain that future weeks won’t always be so lopsided in StarsDraft’s favor. Sign up at StarsDraft.com and check out our week 15 salaries for yourself.