I'm going to admit my bias: I'm a huge fan of showing your work. Baseball is by far the most advanced sport in the analysis department, and a lot of those advances were made because people put their methodology in the public domain. And I think hockey benefits in the same way.
So when the guys at Power Scout hockey asked for feedback on their player valuation system a month ago, I thought it was worth getting them to expose the details of their efforts to everyone. They've come out with some details of what they found to be important, and it's worth everyone having a look to see what they've done. (Goaltending details are here.)
My comments so far – and feel free to add to them:
First, for goalies:
1) Goaltending, at 27%, is vastly over-valued. If this were in any way true, goaltenders are underpaid by 50-75%. We'd expect to see a lot more goalies paid the league maximum.
2) Somehow team shots for worked their way into the goaltender valuations. We know goalies don't drive their own team's offense.
3) "Strength of Opposition" – the claim here is that stronger opponents (presumably ones that score more goals?) generate more difficult shots. This is not particularly true at even-strength, and to the extent that it's true on the PP, a team's penalty-drawing abilities are important, not their goal totals.
4) Short-handed goals against make an appearance in the goalie valuation too. Forget the predictive value of this quantity – is it sufficiently regressed for the tiny number of shots a team faces when it's on the PP?
And for skaters:
1) Where are faceoffs? That's a true talent for centers
2) There are a few claims here that seem really wild: shooting percentage doesn't matter for centers but is a huge deal for wingers? Hits (which have never been found to be important by anyone) are hugely important for centers, but not really for other players? Games played matter for centers and defensemen, but not for wingers?
3) How is scorer bias handled for turnovers and blocked shots? Do they have information about the player who received a giveaway or had the puck taken away from him?
4) How can turnovers be a big deal for centers, but not for wingers? The problem is that players don't necessarily always line up at the position that's listed on their team's website. We have actual location data that can be used to determine whether a player turned the puck over at a location that might result in a goal.
5) Skating rating – sounds subjective. What is it?
6) There's a line about how assists by centers are the least meaningful, followed by those by wingers and then those by defensemen. I'm again not totally sure what this means, but why not further differentiate between first and second assists?
I'm not really sure what the methodology is at this site, but it seems like a bunch of variables got dumped into the regression hopper and we're seeing a statement of the regression coefficients. I'm not a big fan of that method either because I think it leads us to some of the strange results we've seen here. Further comments are appreciated by the site owners, I'm sure.
