I last updated the statistical analysis FAQ a year and a half ago, and the analytical view of hockey has shifted profoundly in that time. So I wanted to add answer to a few more questions now that we've had time to digest what we've learned.
First: luck. This seems to be one of the most controversial and infuriating topics in hockey, and I've been advised to call it something else, like "random chance" or "random variation" that people with a reflexive hostility to the role of luck in sports will find more palatable. But 'luck' is exactly the right word for what happens – when a bouncing puck goes over a player's stick or a deflected puck goes right in the top corner of the net – and I think everyone who plays attention to hockey knows that luck drives outcomes. All we disagree on is the extent to which luck is responsible for those outcomes.
Now some people might ask why we should expect luck to show up in the final analysis. After all, don't the lucky bounces even out in the end? Over the course of a half-dozen seasons, yes, but not in the short-term – the short-term being an entire season. A team's actual goals for and goals against are merely a sample of their ability to score and prevent goals, and we need a lot of games to see their actual talent. One of our commenters suggested a great analysis that an industrious reader could do at home: watch all of a team's goals for and goals against over the course of a season, and count how many were lucky – you'll get an estimate of how much of a role luck plays in goal-scoring, and you'll be able to figure out how much that team benefited (or not) from luck. I can guarantee you that if you watch a few hundred goals on tape, we'll make a believer out of you.
At any rate, this topic has been explored in great detail over the last year, and I'd like to offer up a number of articles, here and elsewhere, that explain the role of luck in the game. A common analysis technique is to start with equally-talented teams or players and simulate them in what amounts to a weighted coin-tossing experiment. The difference between the results of this experiment and actual NHL outcomes is the skill component in the league. Some great examples:
- The role of luck in team winning percentage
- The role of luck in team shooting percentage
- The role of luck in special teams shooting percentage
- Repeatability of Special Teams Performance
I think the principle behind these experiments is indisputable. If we had a league full of average players and teams, we'd see a certain distribution of outcomes solely because we have a limited number of games to observe their performance. And because we actually have a lot of stars and a lot of scrubs – and they're not evenly-distributed across NHL teams – we actually see a much broader distribution of outcomes because of all that skill.
Some of the conclusions of this analysis may be surprising, in particular that there's very little in the way of goal-scoring talent at the team level. If you dispute this, I'd suggest you go through our goal-observation experiment. You won't necessarily get the same luck estimate as we get over thousands of goals over multiple seasons, but you'll see that shooting percentage is by no means all skill.
