Mike Lehr's Blog

I ran across a good article by Malcom Gladwell in the February 14 & 21 issue of The New Yorker titled, “The Order of Things.” The detail with which he explores rankings of colleges, hospitals and cars demonstrates the immense subjective potential rankings have. What is even more astounding is Gladwell’s discovery of the degree to which many organizations hold their leaders accountable for their place in these rankings.

From an intuitive perspective, people tend to have an emotional connection to statistics; they satisfy feelings for certainty, clarity and knowledgeableness. Thus, when we express arguments statistically, they tend to carry more weight than if we simply express them in words. Rankings clearly define for us what is best, better and good. However, they are more akin to magic where reality is but a trick. Thus, the feelings we receive from rankings (certainty, clarity, knowledgeableness) are satisfied because we want to believe their magic is real.

As a rule, unless the ranking is comparing very similar things against a single, measurable criterion, it is highly subjective. Therefore, here are some important questions to ask about the ranking to discover how its trick works:

• Is it really comparing similar things?
• Is the ranking based upon multiple criteria?
• How important is each criterion and is it valid?
• How does it weight the criteria?
• Is it using some criteria as proxies for things that are difficult to quantify or research?
• What important criteria are absent because of these difficulties?
• Is the difference between one rank and each of those immediately above and below it that significant?
• How accurate was the data collected for each criterion?
• What problems might have retarded data quality?

Applying these questions will demonstrate that our affinity for rankings is more emotional than pragmatic.