Why Per-Game Metrics can't Evaluate the Draft

Earlier this week, I briefly discussed Malcolm Gladwell's assertion that there's no relationship between draft position and a quarterback's performance in the NFL. Gladwell cited a paper that showed that the per-play performance of quarterbacks drafted 100th or later was the same as that of quarterbacks drafted 1-10 overall.

Gladwell's claim is easily-debunked, but I wanted to give you an example of what kind of wrong results you get when you engage in these kinds of shenanigans. Let's look at NHL entry drafts from 1979-99 - the brown line shows the percentage of draft picks who eventually play at least one game in the NHL, while the blue line shows the average number of points scored per NHL game scored by those forwards and defensemen who played in the NHL:

Draft7999_medium

If you look at the per-game performance, you would conclude that beyond the 20th pick or so, there's no difference between the drafted players. 250th picks who played in the NHL scored just as many points-per-game as 25th picks who played in the NHL. Hence, NHL general managers are terrible evaluators of talent outside of the obvious two dozen future stars. Time for me to drop a 'QED' and get my findings published in a reputable academic journal!

The only problem is that most late-round draft picks never play in the NHL. So the average played drafted 25th provides some value to a team at the NHL level, while the average player drafted 250th probably never gets invited to an NHL training camp. So one high-reward pick in the 9th round - say Henrik Zetterberg - makes up 90% of the per-game statistics and makes all of the guys who never played in the NHL look a lot better. (Or as they say of Bill Gates: when he walks into a room, everybody's average net worth goes way up, but the percentage other than him who billionaires did not...)

The correct conclusion is that general managers do a good job of evaluating draft-age talent. Sure, there's some selection bias in our chart - first-round draft picks will get many more chances to establish themselves - but over the course of 20 years, players get drafted in the order of how they'll ultimately perform at their peaks. A monkey at a dart board absolutely would not produce better results.

X
Log In Sign Up

forgot?
Log In Sign Up

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

Join Arctic Ice Hockey

You must be a member of Arctic Ice Hockey to participate.

We have our own Community Guidelines at Arctic Ice Hockey. You should read them.

Join Arctic Ice Hockey

You must be a member of Arctic Ice Hockey to participate.

We have our own Community Guidelines at Arctic Ice Hockey. You should read them.

Spinner.vc97ec6e

Authenticating

Great!

Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.

tracking_pixel_9355_tracker