Earlier this week, I briefly discussed Malcolm Gladwell's assertion that there's no relationship between draft position and a quarterback's performance in the NFL. Gladwell cited a paper that showed that the per-play performance of quarterbacks drafted 100th or later was the same as that of quarterbacks drafted 1-10 overall.
Gladwell's claim is easily-debunked, but I wanted to give you an example of what kind of wrong results you get when you engage in these kinds of shenanigans. Let's look at NHL entry drafts from 1979-99 – the brown line shows the percentage of draft picks who eventually play at least one game in the NHL, while the blue line shows the average number of points scored per NHL game scored by those forwards and defensemen who played in the NHL:
If you look at the per-game performance, you would conclude that beyond the 20th pick or so, there's no difference between the drafted players. 250th picks who played in the NHL scored just as many points-per-game as 25th picks who played in the NHL. Hence, NHL general managers are terrible evaluators of talent outside of the obvious two dozen future stars. Time for me to drop a 'QED' and get my findings published in a reputable academic journal!
The only problem is that most late-round draft picks never play in the NHL. So the average played drafted 25th provides some value to a team at the NHL level, while the average player drafted 250th probably never gets invited to an NHL training camp. So one high-reward pick in the 9th round – say Henrik Zetterberg – makes up 90% of the per-game statistics and makes all of the guys who never played in the NHL look a lot better. (Or as they say of Bill Gates: when he walks into a room, everybody’s average net worth goes way up, but the percentage other than him who billionaires did not…)
The correct conclusion is that general managers do a good job of evaluating draft-age talent. Sure, there's some selection bias in our chart – first-round draft picks will get many more chances to establish themselves – but over the course of 20 years, players get drafted in the order of how they'll ultimately perform at their peaks. A monkey at a dart board absolutely would not produce better results.
