Fantasy sports has its roots in Major League Baseball. The first rotisserie leagues were established in 1980, played between journalists at a New York City restaurant named ‘La Rotisserie Française’.
The game of baseball generates reams of data by its very design — there are several repeated trials of batter-vs-pitcher per game and hundreds of games per season. Analysts argue the relative virtue of this or that new metric to capture the true value of each play or to measure the skill of each player. Several full length Hollywood films focus on the application of data science to the MLB by professional scouts and teams looking for an edge. For data-driven sports analytics, MLB is the Holy Grail.
StarsDraft.com has been offering real-money contests since September of 2014. In these past seven months we’ve designed, developed and launched four unique daily fantasy sports products.
For every new sport we try to capture the fundamental elements of the underlying game and put out a product that resolves at least a few of the main issues. Across all sports we’ve increased positional flexibility and produced scoring systems that are as closely tied to athlete performance as possible. Unique solutions — like the SUB position in NBA, or Pick 7, Play 5 in PGA — aim to alleviate some of the more frustrating aspects of these sports; late-swap and the Friday cut, respectively.
For MLB we believe we’ve made similar strides. Some of the highlights include:
- Pitching Staff – Users will draft a Pitching Staff instead of a single pitcher. All points accumulated by any pitcher on the corresponding team will count towards the score of the Pitching Staff.
- Pinch Hitter – Only the top 7 of 8 batters drafted will count towards a lineup’s final score.
- Positional Flexibility – Rosters will consist of one Pitching Staff, 3 outfielders, 4 infielders and one Pinch Hitter (any non-pitcher).
- Performance-Based Scoring – Scoring will be based on the real-world value of each play as determined by the run-expectancy matrix.
The scoring rules will be as follows (referred to as performance scoring from here on):
- RBI / Run Scored: 1
- Base Hit / Walk / Hit By Pitch: 1.25
- Double: 2
- Triple: 3
- Home Run: 5
- Stolen base: 1
- Caught Stealing: -1
- Innings Pitched: 2.25
- Strikeout: 0.40
- Earned Runs: -2
- Hit or hit equivalent (hit, walk, HBP): -0.25
- Shutout: 1
- No Hitter: 2
- Complete Game: 2
- Perfect Game: 5
At a high level this product is designed to be engaging throughout the entire baseball game, allows users maximum flexibility when making roster decisions, provides relief from the inherent inconsistency in MLB batting performance, and awards sensibly weighted fantasy points that correspond as closely as possible to the real-world value added of each statistical category. What follows is an explanation of the thought process that leads us to each of these decisions, and the data that backs it all up.
Before going into how we wanted StarsDraft.com ’s MLB product to function, we needed to understand the current offerings in the industry. There are two dominant products in the market today and we will refer to them as “basic” and “bonus” scoring. Both require owners to roster a complete infield (1B, 2B, 3B, SS, C) and 3 outfielders. Basic scoring has a single pitcher and bonus scoring has a two-pitcher setup. The scoring systems of each are outlined below:
- Win: 4
- Earned Run Allowed: -1
- Strikeout: 1
- Innings Pitched: 1
- Base Hit / Walk / Hit By Pitch: 1
- Double: 2
- Triple: 3
- Home Run: 4
- Run Scored: 1
- Run Batted In: 1
- Stolen Base: 2
- Out: -.25
- Innings Pitched: 2.25
- Strikeout: 2
- Win: 4
- Earned Run Allowed: -2
- Hit against / Walk / Hit Batsman: -.6
- Complete Game: 2.5
- Complete Game Shutout: 2.5
- No Hitter: 5
- Single: 3
- Double: 5
- Triple: 8
- Home Run: 10
- Run Batted In: 2
- Run Scored: 2
- Walk / Hit By Pitch: 2
- Stolen Base: 5
- Caught Stealing: -2
Obviously these two products are very different. From a player’s perspective, diversity in products is a good thing. As with all StarsDraft.com products we needed to decide which aspects of these two established systems work well, what could be improved, and what we could reinvent entirely.
The first major innovation of the StarsDraft.com MLB product is the concept of a Pitching Staff. Owners will draft a single team’s pitching staff instead of a starting pitcher and will receive all points awarded to any member of that staff throughout the entire course of the game.
In season-long formats, owners draft one or two starting pitchers. Stats from various categories (wins, saves, strikeouts, earned runs, etc.) are accumulated throughout the season and, in the most popular rotisserie style leagues, a fantasy MLB team’s performance is determined by how it ranks across several of these categories at the end of the season. The prevailing daily fantasy (DFS) baseball offerings mirror season-long leagues relatively closely. Users draft one or two pitchers and are awarded points based on their in-game stats.
The translation from season long to daily sounds straightforward enough, so why change it?
As we’ve seen with other sports, compressing the rules and scoring systems of a season-long format into a daily game is not necessarily the optimal way to play DFS. Many aspects of season-long sports work well only when games are played repeatedly over the course of a full season. These same features make less sense when put into the context of a single-day contest. In the modern era, where pitchers routinely pitch 6 or fewer innings, statistics like the Win are an increasingly suspect measure of starting pitcher performance. The minimum requirement for a pitcher to earn a Win is 5 innings pitched. In the early 1900s, when the Win statistic originally caught on, pitchers routinely pitched entire games —pitching at least 5 innings was nearly a given outside of injuries or games being called early.
The game has since changed. Over the last 3 MLB seasons, 16% of starters pitched 4 innings or fewer; they weren’t even eligible for the Win. (For an excellent history of the win statistic check out, “Origin of the Modern Pitching Win,” by Frank Vacarro, originally published in the Spring 2013 Baseball Research Journal.) Needless to say, having your starting pitcher pulled after 4 innings makes for a very uneventful afternoon in the current win-bonus driven DFS MLB offerings. Perhaps more frustrating from a fantasy perspective is that the Win is frequently determined by factors unrelated to the play of the starting pitcher. Maybe the bullpen pulled out an excellent save, or the offense caught fire and won in a high-scoring, poor-pitching shootout. The net result of all of these factors is that on any given day the Win bonus is largely random. Below is a plot of earned runs allowed for starting pitchers (who pitched at least 4 innings) awarded a Win vs. those not awarded a Win:
Pitchers awarded the Win allow fewer runs per inning on average. But these distributions are more similar than they are distinct — many strange exceptions exist between the winning and losing distributions. Of the roughly 14,581 games analyzed, 1,920 of them had starting pitchers with an ERA of exactly 0.0 — they pitched as well as they possibly could have in the innings they played (as measured by earned runs allowed). Of those 1,920 ERA 0.0 performances, 300 were ultimately losing pitchers (15.6%) and didn’t receive the Win. More telling is the 30% of losing pitchers that threw for an ERA of 3.0 or lower. If sustained for an entire career, a 3.0 ERA would rank the pitcher in the top 80 MLB pitchers of all time! Still, despite such an excellent showing, these pitchers were not awarded a Win.
To understand the impact that all of this has on fantasy performance, we compared fantasy points in the three scoring systems for two types of pitchers: 1) Pitchers with an ERA of 3.0 or better that did not receive a Win bonus versus 2) Pitchers with an ERA over 3.0 that did receive the bonus:
The overlap of these two distributions indicates the extent to which each scoring system differentiates good and bad play at the mound. For basic scoring the distributions have very similar means: pitchers who throw 3.0 or better ERAs on losing teams have fantasy scores very similar to those who throw worse than 3.0 and get the Win. Bonus scoring has slightly more distinct distributions, indicating less emphasis on the Win in their scoring rules. Performance scoring eliminates the Win bonus entirely and as a consequence does the best job of the three systems at differentiating between good and bad pitching.
But there’s a very important catch. In a one or two pitcher roster setup, the Win bonus must be preserved in order to keep team owners entertained throughout the course of a full 9-inning game. Without a Win bonus owners have little to root for once the starting pitchers have left the mound. So we found ourselves in a situation where the Win statistic was clearly a poor statistic for scoring fantasy pitching performances. But getting rid of the Win would take the fun out of the final, critical innings of most ball games.
Our solution? Set up a pure scoring system and allow owners to draft a pitching staff that earns stats for the starter and the bullpen. No need for a win bonus. No more starting pitchers getting pulled after 4 innings, ruining lineups and frustrating owners. Winning teams will be those on which the starter-bullpen combination pitched the best possible game, and fantasy points will reflect that performance. Best of all, every pitch in the game will matter.
With NBA we introduced the notion of a SUB position to account for the late-swap issue. For PGA we allowed users to pick 7 golfers and play 5, thus increasing the likelihood of having a complete roster of 5 golfers after the Friday cut. When we asked our users, they overwhelmingly wanted some similar roster construction in our MLB product. The Pinch Hitter behaves more like PGA than NBA; the salary of the Pinch Hitter counts against the overall salary cap. A roster is scored by totaling the top 7 of 8 batters, and the Pinch Hitter will fill in for the lowest scoring batter he outscores.
Baseball is one of the highest variance daily fantasy sports. Ignoring the very real threat of entire games being canceled due to weather, on any given night an all-star batter can easily go 0/4 and put up 0 points. An average batter has an on base percentage of around 32%.This means that in four at-bats they have a 21.4% chance of going0/4: (1- OB%)^4 = .214. Given an 8-man roster the probability of having at least one batter with 0 points can be calculated as 1 – (1 – probability of going 0/4)^8, or 85%! In other words we would expect that 85% of 8-man teams would have at least one batter with 0 fantasy points. In a pick 8, play 7 Pinch Hitter roster configuration, the odds of having at least one batter with 0 points in your highest scoring 7 is still relatively high — roughly 55%. So the Pinch Hitter solution relieves a bit of stress of regular busts in fantasy MLB, while preserving the difficulty inherent in the game.
Outside of a Pitching Staff and Pinch Hitter, we needed to determine what sort of roster configuration we would use. StarsDraft.com polled active players in the community and received a ton of great feedback. The ultimate decision was that the low-level traditional baseball fielding positions simply don’t make sense in the context of a fantasy sport that only awards points for batting and pitching. Why force our users to draft catchers, short stops and 3rd basemen if they think a team of 1st basemen offers the most value? Like hockey, where we discarded the need for the Right Wing/Left Wing / Center distinction, in baseball we decided to allow owners to draft 4 infielders and 3 outfielders (outside of the Pitching Staff and Pinch Hitters, which can be a player in any non-pitcher position). This setup offers a large degree of flexibility, increasing the fun and challenge of roster construction, without the undue burden of selecting to fill fielding positions that are largely irrelevant to fantasy performance.
Scoring and the Run-Expectancy Matrix
Given the addition of a Pitching Staff and Pinch Hitter, we needed to pay special attention to the scoring rules, making sure to strike appropriate balance between hitting and pitching scoring potential. We started by looking at the incumbent systems. Basic scoring employs fewer statistical categories and the points awarded for each statistic are relatively simple to grasp: one point for a single, two points for a double, etc. In terms of the pitcher/batter balance, approximately 43% of all points are scored by pitchers in basic scoring.
For bonus scoring the relationships between the different statistical categories is more complicated, and there are bonuses applied for single-game accomplishments such as pitching a no-hitter or a complete game. 26% of all points are scored by pitchers in bonus scoring, but users must draft 2 pitchers to complete their rosters, making the fraction of the average roster’s scoring that comes from pitchers somewhere around 50%.
One interesting thing to note about both of these systems is the different relative values between statistical categories. In basic scoring, a home run is worth 4 singles, runs, or runs batted in. In bonus scoring a home run is worth 3 ⅓ singles, 5 runs, or 5 runs batted in. In both systems a stolen base is ½ of a home run! Getting caught stealing, on the other hand, is only a 2 point deduction in bonus scoring and does not count at all in basic scoring, and so on. Not only do the two systems use different statistical categories, the weights between the categories do not agree. In other sports this sort of conflict would require us to dig into the data in order to figure out which system was more pure; which one more closely tracks the real-world value of these plays? Thanks to the sabermetric revolution in baseball we have plenty of pre-existing research on which to base our analysis.
One of the key tools of sabermetricians is what’s referred to as the run-expectancy matrix. This matrix tells us the expected-runs value of every base-runner/out circumstance. Below is the matrix for 2014:
The purpose of the matrix is to better understand and ascribe value to different baseball situations. Coaches, scouts, and baseball nerds across the country use the run-expectancy matrix to assess in-game decision making, identify prospective talent, win bar bets, etc. It’s not perfect, and the expectancy values shift slightly from season to season, but this matrix is perhaps the best guide we have to understand the value of any particular baseball circumstance.
The key idea behind the run-expectancy matrix is to tie every situation to a single measure of value — in this case the value of measure is ‘expected runs in an inning’. By framing the various base-runner/out circumstances in terms of expected runs we’re able to compute the value of baseball actions as transitions between circumstances (or ‘states’). With 0 outs and the bases empty, for example, the expected runs are .4552. If the next batter hits a single, the expected runs increases to .8182 — a total added run-expectancy value of .8182 – .4552 = .363. A single in this circumstance is worth roughly .363 additional expected runs. Every baseball play can be viewed as a transition between two run-expectancy states in this way. To visualize this concept and assist with the production of StarsDraft.com’s MLB scoring system, I created the a run-expectancy transition matrix, showing all of the possible state transitions and their run-expectancy impact. Red is a loss in run-expectancy, blue is a gain:
To read the matrix, pick any state on the left and follow that row until it intersects with the column of any corresponding state on the top. The run-expectancy value of transitioning between the left state and top state is what determines the color of the square which represents that transition. For example, if the first (left) state is ‘0 Outs – Bases Loaded’ and the next (top) state is ‘2 Outs – Runners at 2nd and 3rd’ we find the intersecting square and determine from its orangish color that this transition represents roughly a 1.5 run-expectancy loss (as indicated by the color scale shown on the right of the chart).
Armed with this tool to measure the value of plays directly, we can start to construct a performance-based fantasy scoring system. The value of measure, from which the rest of the scoring system will follow, is a bases empty home run. On a bases empty home run a team earns a 1 run and ends in the same base runner/out situation they began in (i.e. there are no complicating factors like base positions or RBIs to consider). For convenience we let this value be worth 5 fantasy points — it’s an easily computable number that gives us room for smaller point denominations for less impactful plays. All other baseball actions can then be translated into their fantasy-value equivalent using this simple formula:
run-expectancy value of the play * fantasy value of a run = fantasy value of the play
A single / walk / hit by pitch then is worth .25 * 5 = 1.25 points (.25 is determined by taking the average of the value added by for a single across the 3 different outs states, and rounding to keep things simple). A double is worth roughly a .4 increase in average run expectancy (again, averaging across all the possible outs states): .4 * 5 = 2 points. A triple is roughly a .6 run-expectancy increase: .6 * 5 = 3 points. And, as already mentioned, a home run is worth 5 points.
The method here is more or less the same for determining the value of more nuanced stats like scoring a run or batting in an RBI. The value of an RBI in terms of run-expectancy is
1*(run added) – (run-expectancy generated from the hit) – (run-expectancy before the hit)
averaged across all base / out circumstances. When you run the math, an RBI comes out to a roughly .4 increase in run-expectancy on average (2 fantasy points). The attribution of those points then needs to be split between the batter (RBI) and the base-runner (Run). As it turns out the odds of getting on base are roughly equivalent to the odds of scoring a run once on base (about 30%), so both a Run and an RBI come out to approximately 1 point. (Conversely, an earned run for a pitcher will be worth -2 points)
Last but not least, we needed to appropriately account for steals. To figure out how much steals were worth we returned to the expectancy delta matrix. Successfully stealing 2nd increases run-expectancy by .15 runs, stealing 3rd by .18 runs, and stealing home by .4 runs. Taking the weighted average of these figures based on how frequently each base is actually stolen gave us the value of an average stolen base: roughly a .2 increase in run-expectancy (or 1 fantasy point, slightly less than a single). We wanted to capture the realistic risk-reward tradeoff for attempting a steal by appropriately accounting for the run-expectancy value lost when runners are caught stealing. Interestingly, getting caught stealing entails a larger loss of run-expectancy than the run-expectancy gained from successfully stealing a base (about a .3 loss in run-expectancy on average, or 1.5 points). An owner of a base-runner who gets caught in the act of stealing already experiences their own loss of expected fantasy points scored due to the loss of a future possible run. Therefore, we reduced the penalty for getting caught stealing from -1.5 points to -1 point.
We wanted StarsDraft.com ’s scoring system to award an average of around 35% of points to pitching staff, slightly increasing the roll of batter scoring compared to other scoring systems. To quantify the value of the major pitching statistics we returned to the concept of run-expectancy. MLB teams score an average of around .45 runs per inning. If we assume that each out is preventing ⅓ of the runs expected to be created in any given inning, each out is then preventing ⅓ x .45 runs = .15 runs. Multiplied by 5 points per run, that comes out to .75 points per out, or 2.25 points per innings pitched.
It was a bit more nuanced to figure out appropriate points to subtract from pitchers for runs allowed vs. hits. Obviously, most pitching staffs will pitch a full 8 or 9 innings, or 24-27 outs. We wanted to avoid unduly rewarding extra innings and ensure sure that all things being equal performance, as measured by stats like ERA and/or WHIP (walks plus hits per inning pitched), was the major determinant of fantasy points earned.
Our logic for pitcher scoring is as follows. In order to maintain our target 35-65 split between pitcher and batter scoring we needed an average inning to be worth around 1 point (before strikeouts and full-game bonuses). Average WHIP is around 1.3, and as mentioned earlier, average runs scored in an inning is roughly .5. Preventing runs is the ultimate goal of a pitching staff, so it needed to be the most heavily weighted of these two entangled statistics. Furthermore there are circumstances when walking a batter or allowing a base runner to advance makes sense in the context of the game, the same can not be said for allowing runs to score. Setting earned runs to -2 points, as discussed earlier, and all hits / walks / hit batsmen to -.25 points accomplishes our target balance with reasonable weights between the categories.
So we have +.75 points per out, -2 points per earned run, -.25 points per hit / walk / hit batsman. But what about the all-important strikeout? Trying to uncover the true value of a strikeout was one of the most enjoyable parts of developing StarsDraft.com ’s MLB product. Why is a strikeout more valuable than a ground out or fly out?
It stands to reason that the true value added by securing a strikeout as opposed to getting a batter to hit into an out is the value of preventing a fielding error opportunity. The average errors per game since 2010 was approximately 0.6. The average number of hits per game was roughly 8.5. Doing some very, very rough math the chance of an error on any given hit is somewhere near .6 / 8.5 or ~ 7%. The value of preventing an error opportunity can be measured as the average run-expectancy cost of committing an error multiplied by the probability that an error will be committed on any given hit.
While getting data about the run-expectancy cost of errors is difficult, we can use the run-expectancy transition matrix to put some firm boundaries on the problem. An error must be worth somewhere between .1946 runs (the least costly negative state transition: a ‘2 Outs – Bases Empty’ error that results in a ‘2 Outs – Runner At 1st’ state instead of retiring the side) and 4.082 (a ‘2 Outs – Bases Loaded’ error resulting in 4 scored runs and ending in a ‘2 Outs – Bases Empty’ state, rather than retiring the side). Depending on how frequently certain types of errors actually happened, the weighted average cost of an error could swing dramatically.
On average, allowing at least one error corresponds to a 1.14 increase in expected opponent runs scored. Using this as a rough estimate of the run expectancy cost of an error, we can estimate the value of a strikeout as
7% errors * 1.14 runs per error * 5 points per run = .4 points per strikeout
To round out or MLB scoring we decided to include full-game bonuses for pitching staffs. This may seem like a departure from our standard no-bonus policy, but bonuses themselves were never the issue. It’s arbitrary bonuses we try to avoid — awarding points for accomplishments that have no apparent tie to player performance or are otherwise already encompassed in the standard scoring rules. In baseball there is real value added by pitching staffs that is unaccounted for during full-game performances like shutouts, no hitters, and perfect games (as there is in hockey where we award a shutout bonus to goalies). These accomplishments are varying degrees of pitching perfection — they allow a team to make a different set of strategic decisions throughout the course of the game that ultimately impact game outcomes. Complete games are impressive, single-individual efforts that benefit a team above and beyond the boxscore by allowing the bullpen and rest of the staff to take the night off. To reward these unaccounted for accomplishments we will award the following bonuses: 1 point for a pitching staff that throws a shutout, 2 points for a no hitter, 2 points for a complete game, and 5 points for the rarest pitching accomplishment of all — the perfect game.
If you made it this far you’re a true DFS MLB die hard, congratulations. You should try out StarsDraft.com for yourself to see how all of this nonsense translates into a living, breathing MLB product. The new innovations that include the Pitching Staff and Pinch Hitter, combined with increased positional flexibility and a performance-based scoring system make StarsDraft.com ’s MLB product the most unique in the industry.