You're probably familiar with Malcolm Gladwell's lazy analysis: he's wrong about 'The Birthday Paradox' and he was wrong when he claimed that NFL GMs have no drafting skill. This time, he jumps into something we're very familiar with in the sports department: making up rankings. You can't see his New Yorker piece for free, but here's a summary of his piece on college rankings.
If you’ve ever tried to make up rankings, you’ve probably sat down with your final list and looked to make sure everything seemed right. If you find Zdeno Chara or Albert Pujols or LeBron James in the bottom half of your list, clearly there’s a bug in your ranking system and you go back to the drawing board. Well, for Gladwell, getting things wrong is a feature, not a bug:
"One of the techniques Gladwell deploys in the article is to reorder certain schools by changing some of the metrics used to rank them. He revises a top ten list of schools by doing what U.S. News doesn’t—including the cost of tuition as a variable. U.S. News would include cost if value for the dollar was something it judged important. Simply taking cost into account, the rankings of seven of the top schools changed and three schools—Northwestern University, Columbia University and Cornell University–dropped out of the top ten, replaced by the University of Alabama, the University of Texas, and the University of Virginia. Concludes Gladwell: "The U.S. News rankings turn out to be full of these kinds of implicit ideological choices.""
This is not an ideological choice – Gladwell is so enamored of his own contrarianness that he thinks being wrong makes his system better. From what I could find on the internet, the average SAT score at Cornell would be in the 93rd percentile at the University of Alabama. The student-to-faculty ratio at Alabama is 20:1; at Cornell, it's 11:1. Those two facts aren't the be-all and the end-all of a college experience, but Gladwell is concerned with academics, where they certainly are a good indication that Cornell > Alabama.
When you're creating a fixed size group, we know population size is very significant, and the University of Alabama simply draws from a smaller potential student population than Cornell, in addition to having a much smaller endowment. Even if it offers better value-added per dollar for many students, it takes a giant leap of faith to think that it's a better school. North Dakota may be much more hockey-crazy than California, but it won't be long before there are many more professional hockey players from California, because it has 70 times more people than North Dakota!
It's a sad commentary on the state of media that the "premier" data analyst in the popular press is a complete failure when it comes to doing what he's so famous for. Maybe with every major mistake, people are less likely to automatically believe him…But for some reason, I doubt it…
