
University Rankings are Awful. Can We Stop Doing the Same Thing With Countries? - pavel_lishin
http://rottenindenmark.wordpress.com/2013/09/10/university-rankings-are-terrible-now-can-we-stop-doing-the-same-thing-with-countries/
======
crazygringo
Obviously rankings such as these are highly simplistic, and I think it's
pretty obvious that "there is too much noise to say that Yale is better than
Princeton".

But sometimes progress can only be made by focusing on "simple" indicators --
things like childhood mortality rates, whether applied to a country or to a
single hospital.

For a college to decide it wants to go from #200 to #190 in a particular
ranking is meaningless, but going from #200 to #20 is going to be meaningful.
Likewise the president of a country saying, "it's atrocious that we're one of
the 20 most corrupt, or red-tape-laden, countries" \-- it can provide the
necessary impetus to change and improvement.

These rankings are certainly not "entirely unsubstantiated" \-- they're
certainly very far from perfect, but don't throw the baby out with the
bathwater.

~~~
Aqueous
Right. The rankings aren't the last word on an institution. They aren't even
close to it. But that doesn't mean they're meaningless. Similarly with
countries. These metrics refer to real data corresponding to the actual, real-
life experience of people living in these places. If the number of emergency
rooms per capita in a country is low, for example, that corresponds to
actually higher likelihood that you'll be unable to get to one if you live in
that country and are in an emergency situation. The data is real - the author
might not like that it seems like a simplistic representation and glosses over
a lot of nuances, but then why else do we measure and quantify these trends
than to tease out meaningful information from an otherwise incomprehensible,
highly complex system?

Even though a single year snapshot of college rankings isn't completely
meaningful by itself, you can glean a lot from the movements in the rankings,
even with small movements. If a college goes from 200 to 190 in a single year,
you can probably determine precisely what changed how by looking at the
changes in the internals that they used to derive the number. And sometimes
even a small change in ranking does reflect a significant, concrete change in
the experience of students attending these institutions.

Similarly, with countries. The dynamics over time of these metrics is what
gives this data meaning - not any single ranking.

------
WildUtah
Author is getting paid to organize information on a multitude of nations and
their employment and economic prospects and human rights conditions.
Businesses need guidance on where they can get cheap foreign labor to assemble
products without long term problems. Author is the one who is charged with
solving the problem for them.

Then the author decides that businessmen are shallow jerks because they just
want the answer to their problems. They want help to pick a country and
guidance about how to exploit the low wage workers without abusing anyone.
Author thinks they should instead deeply explore the relative conditions and
balance the ineffable cultural and economic differences in the deep way that
only a full-time professional country analyst can. They should be like him.

But the businessmen are paying him to do that so they don't have to.

Perhaps the author would be happier with a nice UN or civil service job where
he's not accountable to anyone for any kind of useful information. Ideally it
would be a job where he can't be fired for self-indulgently coddling his
broad-minded curiosity and refusing to give useful answers. Don't worry
though; he's not the kind of high-minded idealist who refuses to make his
money sending work to the lowest cost country. He's probably an expert on the
authentic third-world experience of getting ladies to do his laundry for him
cheap.

------
ihsw
> I don’t know if high schoolers use college rankings to decide where they
> should get educated.

I've found that some colleges have a reputation centered around
specializations, for example biotech, compsci, civil engineering, and so
forth. Prospective students make decisions based on this reputation, and it's
not difficult to assign a number to that. Whether reputation _means_ anything
qualitative is something else entirely, though.

More relevant, though, is that students also use a variety of other factors in
their decision making. Cost and relative location (ie: distance from home of
closest family, distance to place of work after graduating) are high-priority
factors, meanwhile student body composition (ie: asian vs white) is another.

My point is that companies and people don't make decisions based on a single
and opaque number, it's just one of many variables in determining where to go
and what to do.

The factors I listed above can have numeric values assigned to them, and a
single metric can be derived from a combination of these numbers, however
that's still _somewhat_ opaque.

It may be unfair to assign a number to _an entire country_ , but that doesn't
diminish its value. It would be unwise to use a single metric as input for
decision making, and again that doesn't diminish the value of that metric.
It's just another metric.

~~~
auctiontheory
You have missed the entire point of the article. Arbitrarily quantifying
essentially qualitative measures is misleading (bordering on dishonest) and
potentially harmful, when these made-up numbers are then used to influence
major decisions.

E.g. these bogus college rankings are used to set immigration and other
government policy in many countries: [http://www.theguardian.com/higher-
education-network/blog/201...](http://www.theguardian.com/higher-education-
network/blog/2013/sep/10/university-rankings-influence-government-policy)

The more operations you perform on these made-up numbers, like the
"combination" you mention, the more authentic they might seem. But the result
of averaging a bunch of made-up numbers is just as meaningless, if not more
so.

------
tokenadult
A useful way to compare countries is by revealed preference: where people
choose to immigrate, given a chance. So far the United States fares well in
that kind of comparison, because as much as people complain about the United
States--including people like me who live here--there are many people who
think the overall trade-off of living in the United States is better for them
than the trade-off involved in living somewhere else, even a place where they
grew up and where all their friends and relatives live. Net immigration flows
are pretty strongly suggestive of where people find a good life. International
surveys show a strong preference (not always realized) to live in the United
States.[1] I see a lot more Hacker News threads complaining that immigration
to the United States is too difficult than I see about leaving the United
States being difficult. (Leaving the United States is ridiculously easy, and I
have done that more than once. But even many people who leave to live abroad
eventually come back.)

[1] "More Than 100 Million Worldwide Dream of a Life in the U.S."
[http://www.gallup.com/poll/161435/100-million-worldwide-
drea...](http://www.gallup.com/poll/161435/100-million-worldwide-dream-
life.aspx)

~~~
gdy
And Russia is second in this comparison.

~~~
tokenadult
Where can I find more published information about this issue? Where are the
immigrants coming from?

~~~
rrreese
This was posted on Hacker News a while back, I'm sure there where interesting
comments: [http://peoplemov.in/](http://peoplemov.in/)

Original discussion here:
[https://news.ycombinator.com/item?id=5133098](https://news.ycombinator.com/item?id=5133098)

The rankings for migrant destinations are:

USA 42,788,029

RUSSIAN FED. 12,270,388

GERMANY 10,758,061

SAUDI ARABIA 7,288,900

CANADA 7,202,340

UNITED KINGDOM 6,955,738

SPAIN 6,900,547

FRANCE 6,684,842

AUSTRALIA 5,522,408

INDIA 5,436,012

Data is sourced from the World Bank

~~~
tokenadult
Oh, I see the source at the earlier Hacker News discussion also lists Russia
as a high-ranked country for people moving out. That was my curiosity about
the issue: this town is full of immigrants from Russia.

From that article:

Top emigration countries

MEXICO 11,859,236

INDIA 11,360,823

RUSSIAN FED. 11,034,681

CHINA 8,344,726

UKRAINE 6,525,145

BANGLADESH 5,384,875

PAKISTAN 4,678,730

So, yes, I was speaking of NET immigration, for which the United States still
seems easily to be the top-ranked country. The point in another reply is
correct that laws about immigration limit freedom of movement to some extent,
but that is also revelatory about what people desire. The United States
doesn't have to do anything to keep people already here in, but its government
has been called on by some domestic voters to make it hard for just anyone to
come in from outside.

~~~
WildUtah
And here's just one obvious shouting indication in the data of how complex
reality is. We see that the USA is the number one destination country in
worldwide immigration upthread. Then we see in the parent that Mexico is the
number one country for emigration.

But over the past five years, in every single year, according to the Pew
Hispanic Center, there have been more people moving from the USA to Mexico
than the reverse. And Russia is high up on both lists; that demands some kind
of explanation, too.

You're going to need a lot more detailed information about what is really
motivating migration before you can use numbers like that to measure the
desirability of any nation for residence or citizenship.

------
compare
The reason rankings exist is because often, in the end, a decision MUST be
made. Are current rankings possibly worse than a random draw? - conceivably,
in some cases, but that doesn't have to be the case.

Your concerns about ranking-context and the general difficulty of ranking are,
however, well placed. Solving the ranking problem, for the general case, is a
first-class problem on the same level of difficulty as computer vision or
language processing.

Personally, I think the world deserves a better approach to solving general
ranking.

------
brendano
_Eventually I realized that the only reason the companies pushed so hard, why
they insisted so strongly on rankings and scores over information and
analysis, was because it made it not their problem anymore. They didn’t have
the credentials to pull 50 ‘good’ countries from 100 uncategorised ones, so
they used us to push the responsibility away. ‘It’s not me saying Bolivia is
an 8.2,’ they could tell their boss. ‘A human rights NGO said it was. Making
shoes there is totally approved._

What else would you expect the corporation to do? Eventually, someone in that
organization has to make a decision, and that requires the complex information
about countries' human rights situations to be summarized in a simpler way --
for which a ranking is one approach.

It sounds like they wanted the NGO to do this summarization. If the NGO didn't
do it, then they'd get someone else or from within their organization to do
it.

Something like rankings are inevitable. They fulfill a huge information need.
The fact there are lots of poorly done rankings out there won't change that.
This just means, it's even more important to make _useful_ rankings that
reflect good information.

(And besides rankings, there are perhaps coarser-grained options, like making
several tiers of groupings, only asserting each is an equivalence class ... or
finer-grained options, like giving each country scores in three different
categories. Etc. But in all cases, the incredibly rich and complex source
information has to be summarized somehow.)

------
Jongseong
I share the author's aversion to reducing complex information to rankings. At
the same time, at first reflection it seems that countries would have less
incentives than universities to "juke their stats" (to use the expression in
the article). Of course, economic indicators, crime statistics and the like
are liable to be fudged for political purposes, but my impression is that
countries don't compete with each other on rankings the way universities do.

~~~
rollo_tommasi
While I can't say whether this is true on a larger scale, you may be
interested to learn that Paul Kagame, the ruler of Rwanda, put together a team
specifically for the purpose of maximizing his country's ranking on the World
Bank's ease-of-doing-business index.

[1] [http://www.nytimes.com/2013/09/08/magazine/paul-kagame-
rwand...](http://www.nytimes.com/2013/09/08/magazine/paul-kagame-
rwanda.html?pagewanted=all&_r=0)

~~~
Jongseong
That's a great example. I just hope that at least the changes they made to
maximize Rwanda's ranking on the ease-of-doing-business index were genuinely
beneficial rather than merely cosmetic.

------
wtvanhest
College rankings are foolish.

That said, at least ranking organizations give a real opinion. If your job is
to assess human rights violations, you should numerically assess them.
Qualitative rankings are a weak form of analysis.

Someone has to pick the country, why not the NGO tasked with doing so?

~~~
berntb
I generally agree, it is easy to define good human rights for a country.

But this gets hairy when you start evaluating _how_ bad a non-perfect country
is.

Consider press freedom.

Today, e.g. Turkey and Russia keeps the press down and stop dissenting voices.
_How much_ worse is that than e.g. Sweden, where the press goes in perfect
lock step in certain areas -- few dissenting voices are allowed. (Most of that
is self censure. Who choose what is censored and how? Only journalists know.
Maybe. Part of this is cultural, too. How much? I have no clue.) Note that
this one-way thinking is quite a bit different in the rest of Scandinavia.

Or a very intolerant country might be OK to live in, for a homosexual or
someone of the "wrong" faith. Live and let live, the "different" would just
get lots of curious questions.

Compare that with my present home country, which is a bit homophobic. The only
openly gay guy I've met was a friend of a work friend. He seems to exercise to
be a good street fighter. I _hope_ that is just a coincidence. :-)

~~~
Aqueous
'How much worse is that than e.g. Sweden, '

I'm going to go ahead and say that Turkey and Russia are still, pretty
clearly, worse in this area, and you could probably quantify this. It's not
good that Sweden's press isn't quite as diversified as it could be, but I
think we can both agree that the constitutional protections in Sweden are
objectively better than those in Turkey and Russia.

It might be hard to quantify these differences, but that doesn't mean it's
impossible, and it doesn't mean that attempts to do so are even nearly
worthless.

~~~
Jongseong
Well, according to the Committee to Protect Journalists, today Turkey tops the
world in the number of imprisoned journalists:
[http://cpj.org/imprisoned/2012.php](http://cpj.org/imprisoned/2012.php)

No journalists are jailed in Russia at the moment, but Russia comes in 4th in
the world in the number of journalists killed since 1992 with 56 deaths:
[http://cpj.org/killed/](http://cpj.org/killed/) Turkey is 13th with 21
deaths.

Statistics are indeed useful when we're separating these sort of extremes. The
article gives the example of Norway being less corrupt than Angola, asserting
that you "don't need an index" to tell you that. But comparing Norway and
Denmark is where rankings are less useful, according to the author. The
argument is not that it is always impossible to compare different countries in
quantitative terms.

~~~
wtvanhest
My point is that the author of the paper wants to be paid to work for an NGO
but doesn't believe in actually solidifying their opinion. Listing a bunch of
stuff is not the same as saying this list means you should a) Build a product
in this country, or B) You should not build a product.

Essentially he is being paid for very basic research which is not useful
without an expert in the field. He wants a guy at Big Corp. to make the final
decision so he can wash his hands of it.

I don't support this type of thinking. I would prefer that the author openly
select the best country and stand by his opinion. Maybe it doesn't make sense
to rank country A #132 and country B #135, but it definitely makes sense to
give them a rating in several categories and draw a distinct line.

Or possibly, there is something I don't understand about the business.

~~~
Jongseong
To defend the author's position, I would be wary of providing a single ranking
because the metrics are going to be different for different needs. Say you're
ranking different lodging options at a holiday destination. You can come up
with a single ranking for all the options, whether they are hotels, hostels,
bed & breakfast inns. But it would be meaningless except to weed out the
really bad options unless you knew what the preferences would be in terms of
price, location, atmosphere, decor, etc. Such an index could still be useful,
perhaps, but I see where the author is coming from.

------
aaron695
I be honest, as far as I'm concerned you're in the PR business anyway so roll
with it.

How does removing business from a country help? Will work place health and
safety go up now your employers are getting less money and jobs from overseas?

More people are unemployed, so the child death rate will increase as they
can't afford healthcare.

The concept that companies would chose Norway, one of the richest counties in
the world over one of the poorest is pretty evil.

------
casca
For a different set of rankings created (pulled?) from somewhere else:
[http://www.topuniversities.com/university-rankings/world-
uni...](http://www.topuniversities.com/university-rankings/world-university-
rankings/2013)

Commentary with a British focus:
[http://www.bbc.co.uk/news/education-24024767](http://www.bbc.co.uk/news/education-24024767)

~~~
ravitation
Unfortunately, if the US News' rankings suffer from certain problems, these
rankings suffer much more heavily. These rankings compare universities that
have far more quantitative differences than the US News' rankings; they
compare sets of universities that don't share a language, structure, culture,
etc...

------
adamzerner
I agree that university rankings suck, and I started a website to address it:
[http://www.collegeanswerz.com/](http://www.collegeanswerz.com/).

------
neotoy
What's the matter, tired of ranking at the bottom?

