You Shouldn't Make Facts Out of Opinions: How Much Does Run Environment Affect Defence-Independent Pitching Stats?

A frequent topic of conversation here (and elsewhere as well) is how much better defence independent pitching stats (e.g., FIP, SIERA) estimate (and predict) pitcher performance than more traditional stats (e.g., ERA). However, there may be a major shortfalling to using defence-independent stats because there is an implicit assumption that these stats do a good job of removing much of the random variance that factors into how many runs pitchers actually end up yielding. However, it is important to remember that a stat like FIP is by no means a perfect basis for comparison across players, not only because FIP assumes that all non-HR batted balls are created equally, but also because raw FIP does not account for park and league factors. These factors greatly affect a pitcher's run environment, which can have fairly strong variation on a player's stats.

To account for these problems, a lot of smart folks have adjusted each pitcher's stats to his run environment and then normalized them to a scale with 100 as the league average. However, the scale of 100 may seem arbitrary and does not necessarily provide easily understood information about how good a pitcher is. For example, given a league-average ERA of 4.00, how many earned runs per nine innings should a pitcher with an ERA- of 90 yield? For this reason, we still compare players by the raw statistics (such as ERA, xFIP, etc.). However, this does not solve the problem we have just identified: that there are inherent problems with using those raw statistics because they do not account for run environments. We can talk about how pitching at Skydome so much inflates Blue Jays pitchers run averages but, as long as I've seen, we generally don't bother to actually quantify how much those factors impact a pitcher's run average. Thus, we may actually be over- or under-estimating players across teams.

To determine how each pitcher would do in a neutral context (park and league), I used separate linear models to rescale fangraphs ERA-, FIP-, and xFIP- to actual run averages for each pitcher with more than 100 IP in 2011. Once we have each pitcher's adjusted stats (for example, ERA), we can determine what the park-adjustment was by determining the difference between his park-adjusted stat and his actual stat (in this example, parkadjERA - ERA). Since I did this because I was interested in looking at how much the team a pitcher plays for impacts that pitcher's actual run averages, I grouped pitchers by team and found the mean for each team. As expected, pitchers on the same team tended to have very similar adjustments. Those adjustments are summarized in tabular format and presented for each team by division below (positive adjustments mean that raw stats underestimate pitcher performance, negative adjustments mean that raw stats overestimate pitcher performance).

The first thing that should pop out here is how much more difficult it is to pitch in the American League. Even by ERA, there is only one team for which raw statistics overestimate pitcher performance relative to league average and that team (Tampa Bay) plays in an extremely pitcher-friendly park, yet ERA overestimates their pitchers by just 0.01 runs per nine innings. Along the same lines, xFIP is based almost entirely on league. This should not come as a surprise, since xFIP is designed to adjust for park by normalizing HR/fly-rates to league average. However, since xFIP does not adjust for league, when comparing NL xFIP to AL xFIP, we should penalize NL pitchers about 0.2 runs per nine innings. Although xFIP assumes AL position players are slightly better than NL position players on average accounts for some of the difference, more of it is tied up in the DH. That it estimates pitchers as worth almost 0.2 runs fewer runs per nine innings than designated hitters strikes me as pretty reasonable.

One possible application for this work is to determine how pitchers might perform on different teams: simply add up the adjustments for each team. As an example, what would Brandon Morrow's ERA have been, if he had spent last season with the Padres, rather than the Jays? Well, first we can adjust Morrow's 2011 ERA (4.72) to determine what it would have been in a neutral context by subtracting the Blue Jays adjustment (0.20; 4.72 - 0.20 = 4.52). Next, we add in the Padres adjustment (-0.42) to find that that Brandon's ERA, had he been pitching in San Diego, rather than Toronto, would have been 4.10 (4.52 + -0.42 = 4.10). Now, this method is far from perfect, of course, because park factors really do not do a good job of splitting up differences by pitcher handedness but it does provide at least some rough estimate, which I think is at least a nice starting point.

Thanks to Modest Mouse's "Make Everyone Happy / Mechanical Birds" for today's post title.