Skill-testing question: if a team has an above-average power-play efficiency, what's the likelihood that they're actually an above-average power-play?
Let’s take an actual example from 2009-10 – the league-leading Washington Capitals (25.2% PP) vs the Detroit Red Wings, who were #2 in the league two seasons ago, but closer to average last year:
| PP% | |
| WSH | 25.2 |
| DET | 19.2 |
Washington had 313 power-play opportunities, while Detroit had 307. If we make no assumptions about underlying talent distributions, then the likelihood of Washington's PP being legitimately better than Detroit's is 96%. Do we really believe that?
We already know that you can't assume that an observed result reflects true talent. So we need to regress to the mean. If we simulate the last 22 years of team power-play performance – assuming every team is equally talented – we get a standard deviation of 2.01%. The observed standard deviation is 2.57%, so skill makes up 38% of the total observed performance:
| PP% | Stdev | Luck | Skill |
| 17.9 | 2.57 | 2.01 | 1.59 |
| 62% | 38% |
We can estimate Washington's and Detroit's true talent levels by two different methods – we can simply regress them to the mean by 62%, or we can start from a realistic underlying talent distribution (17.9% average PP%, 1.59% standard deviation) and update the distribution using the observed PP% from 2009-10 via Bayes rule. The results:
| PP% | Actual | Estimate |
| WSH | 25.2 | 20.3 |
| DET | 19.2 | 18.4 |
Now we estimate that there’s only an 80% likelihood that Washington’s PP was better than Detroit’s. The regressed difference between the teams is eight goals vs 20 before the regression. That seems a bit more reasonable, especially since Washington kept their PP unit together over the last two seasons while Detroit lost Marian Hossa and Mikael Samuelsson.
Of course, we know more about these two power-play units than one year's worth of PP%. Detroit played in a tougher division than Washington, and faced tougher competition generally. It's possible that Hossa and Samuelsson were worth one extra win on the PP alone, but Washington's PP is definitely not eight goals better than Detroit's given the same opponents. The key thing to keep in mind is that even what appears to be the best PP in the entire league might not be much above average.
