Brian McNally wrote up a lengthy response to my analysis yesterday, and I wanted to address a few things that remain unclear. I'm not going to discuss Washington's goaltending triumvirate - apparently everyone is very sensitive about whether Michal Neuvirth is a below-average goalie or possibly average.
At any rate, let's take a look at Brian's points:
"in my opinion - the writer goes too far...[in saying] that their PK save percentage can’t "substantially improve" from one year to the next."
Well, if we look at how teams actually do, we find that PK save percentage regresses 85% to the mean and has an R^2 of just 3.5% from one year to another - in other words, PK save % shows very little evidence of being a persistent talent. If we run a coin-tossing experiment at different strengths, we can see how much of a role luck plays in determining team save % over the course of a season:
Most of the skill in save percentage belongs to the goaltender, and in the long-run, goaltenders tend to play no differently relative to league average at even-strength or on the PK. If I was looking for an explanation for Washington's success, I'd start with the fact that they've reduced the number of shots they allow on the PK before I went anywhere near the very luck-driven save percentage.
"The new system is completely different from last year’s passive PK. It uses a deep group of forwards, attacks aggressively, changes personnel relentlessly and – as BTN notes - has limited PK shots by 2.9 fewer per 60 minutes of play than last season. That doesn’t help improve a goalie’s save percentage?"
Again, let's go to the actual data. The R^2 between shot volume at 4-on-5 and save percentage is well below 1% - in other words, there is no apparent relationship between the two of them. As we've seen elsewhere, good defensive teams limit shots against; they don't limit the inherent danger that those shots pose.
"BTN’s writer claims there is no evidence that shot quality has any real effect on save percentage. Absence of evidence is evidence of absence? Conventional wisdom isn’t always right, of course. Statistical advances in baseball have long proved this. But a shot from the point with little traffic in front and a forward hounding you seems different than a one-timer in the slot. I’d love to learn more about that. I’m certainly not averse to changing my opinion. But in baseball we have people charting each play in every MLB game now and using that to devise better defensive metrics. Do NHL games get the same treatment with regard to shot quality? Maybe they do."
Of course there is a difference between a shot from the point in traffic and a one-timer in the slot. Nobody is arguing otherwise. But at the team level over the course of an entire season, there is very little evidence of much in the way of shot quality talent. Given that even-strength save percentage is something like 60% luck and that goaltending accounts for a not-insignificant chunk of the remaining 40%, there just isn't much left for shot quality. The successful teams of the last few years - Detroit, Pittsburgh, Chicago - have dominated shot counts but have done little to improve their save percentage.
Numerous people have been charting scoring chances over the last three seasons. If there were teams that had a higher-than-average shot quality, we'd see this in the scoring chances - they'd have more scoring chances per shot. We don't see that any more than we saw it in analyzing all of the shot location and shot type data that's recorded at NHL arenas.
"If the idea is that the Caps’ PK will regress in the future it’s hard to see how including the third-string goalie is valid."
This misses the point. PK save percentage is heavily luck-driven. We expect it to regress very heavily to the mean regardless of who's in goal.
"Anyway, it’s odd to get the "sample size" argument thrown at me when BTN uses 5-on-3 luck (.938 save percentage) as another reason the PK will dip dramatically. The Caps are 8-for-9 killing 5-on-3s, right? They’ve been down two men for exactly 8 minutes, 54 seconds this season. That, my friends, is the epitome of a small sample size. Four of those two-man disadvantages have been 53 seconds or less. Two were less than 30 seconds. The Caps just don’t go down two men often enough for it to matter. It last happened on Dec. 9."
This has nothing to do with small sample sizes. Washington has somehow had a .938 save percentage when they're down two men. 3v5 save percentage is roughly 80% luck. Even in the small amount of time they've been two mean down, they've allowed three fewer goals than we would have expected them to. There's no reason to expect that to continue in the future.
"Boudreau might think this recent patch of PK success is luck....And he'd never agree that the team's overall PK rate is unsustainable. Maybe not top 3. Top 10? Top 12? He'd be distraught if it fell much lower than that at this point."
I think this underestimates Bruce Boudreau. Everybody in the NHL knows a .938 save percentage at 3-on-5 is unsustainable - it's unsustainable at any other strength, so why would it persist during the most-dangerous stretches of the game? And it's hardly inconceivable that Washington has the 7th-best PK in the league but its observed performance over the rest of the season is 20th in the league. That's what happens when you have a small sample size - sometimes the breaks go the wrong way.