Let's go back to a familiar chart - Jody Shelley's observed shooting talent vs that of Ilya Kovalchuk:
Just a reminder: shooting talent is the amount by which a player exceeds the expected shooting percentage based on the locations he shoots from. I've only included initial shots at even-strength in this analysis - no rebounds, no power-plays.
Looking at that chart, it seems apparent that Jody Shelley could be a better shooter than Ilya Kovalchuk. The probability is quite low - less than 1% - but Kovalchuk has more than a 5% chance of being a net negative shooter, while Shelley has a similar probability of being a net positive shooter.
How is that possible? Anybody who's ever watched Kovalchuk play knows he's a slick offensive player who can pick a corner as well as anyone in the league, while Jody Shelley is the definition of 'hands of stone' - there might be several million people in the world who can shoot better than he can. Those observations aren't wrong - they just aren't perfectly reflected in the limited number of shots taken by the two players.
So how do we resolve this? First, consider every shooter who 200+ shots over the last five seasons and look at the mean and standard deviation of their shooting talent. Next, we simulate a group of league-average shooters over the same number of shots. This allows us to determine the talent and luck involved in shooting:
Avg | Std | Pct | |
Total | 6.36 | 1.53 | |
Luck | 6.36 | 1.03 | 45 |
Talent | 6.36 | 1.13 | 55 |
So we can now start with an underlying distribution for each player - with a standard deviation of 1.13% - and update it based on how the player did in the five post-lockout seasons. We get the following distributions for Shelley's and Kovalchuk's talent:
Now we have a much different story. When we assume an underlying shooting talent distribution, we get an entirely different story - in this case, the likelihood that Jody Shelley is a better shooter than Kovalchuk is about 1-in-10000. And it's probably lower than that - I binned my results in a way that doesn't really give us precision beyond the fourth decimal place. Regardless, it's clear that Kovalchuk must be a better shooter than Jody Shelley.
Here we can see Kovalchuk's observed performance versus his likely talent:
Kovalchuk's true talent level - as implied by five seasons worth of data - is approximately 10.2%, which is 3.4 standard deviations above the mean of our initial distribution. There's a small amount of uncertainty on that z-score, so it's possible Kovalchuk is just (just!!) three standard deviations above the mean. Either way, Kovalchuk is no worse than the second-most talented shooter in the league, and most likely the most-talented. And there's no question that he's better than Jody Shelley.