Changing Goalies = Shuffling the Deck Chairs?

Your starting goalie is getting shelled.  Clearly he's not on today.  So you pull him, and everything calms down…right?  Well, the new goaltender has a much better save percentage:

Note that shot volume against the starter is much higher than usual – so when the goalie gets pulled it is certainly partly his team's fault.  And of course the backup has a better save percentage – the starting goaltender was pulled because his performance that day was a disaster.  On average, his team is trailing by 2.6 goals – close to a blowout – so the game ends up not being contended quite as intensely as if it were tied.  We see two effects here: 1) shot volume against drops significantly – 23%; and 2) shooting percentage actually rises slightly.  So it's difficult to determine whether the replacement goalie's 892 save percentage is due to weaker skills (he's the backup, after all) or if it's simply due to score effects.

One way we can figure this out is to look at the score and time when the goalie was pulled and compare the future goal differential in games played by the backup goalie to the future goal differential in the situation overall:

So what we see here is the disastrous performance of the starting goaltender – putting his team down by over five goals per 60 minutes.  But the interesting effect is that a team's expected goal differential is worse if it changes goalies than if it does nothing.  This isn't surprising – there really is a skill difference between starting and backup goaltenders.  What might surprise many people is that the starting goalie's poor performance is not an indication that his future performance will suffer – in other words, whatever short-term talent variation he appears to have suffered, it does not in general continue past the point at which he'd get pulled.

Now this doesn't mean that changing goalies is a bad strategy.  If your team is losing by 3 goals, the odds of coming back to tie or win is about 10% after the first period, and less than 5% after the second.  Is it worth tiring out your goaltender in a game he's going to lose anyways?  Probably not.  But a team is reducing its odds of a comeback by putting in the backup – just as surely as it increases its odds of losing the game by starting the backup.