
How Reliable Are University Rankings? - ijustlurk
https://arxiv.org/abs/2004.09006
======
qzw
From the abstract:

*>we both formally and experimentally show in multiple ways that this ranking scheme is not reliable and cannot be trusted as authoritative because it is too sensitive to weight changes and can easily be gamed.

That's a feature, not a bug, as far as USNWR (the magazine doing the ranking)
is concerned. A more reliable ranking would be more stable and therefore less
interesting for the universities. Why pay attention to it if Harvard,
Standford, MIT are always 1, 2, 3? It's much more "interesting" and certainly
sells more copies/gets more clicks if the rankings have a certain amount of
volatility. The fact that it's easily game-able also makes it more attractive
to universities because they can do something proactive if they're
dissatisfied with their ranking. All these factors combine to make the ranking
itself seem more important and prestigious.

~~~
HenryKissinger
> The fact that it's easily game-able also makes it more attractive to
> universities because they can do something proactive if they're dissatisfied
> with their ranking

Are you saying that there is nothing lower ranked universities could do to
improve their ranking in a better, more objective ranking of universities?
Because it smacks of elitism.

~~~
jedberg
It's very similar to sports teams.

The tops teams don't vary all that much from year to year. They might change
positions a bit, but it's rare for a new team to show up who hasn't shown up
before.

They essentially have to get lucky and get someone who is super talented into
a long term contract, who then attracts additional talent.

Or they have to invent a whole new way of doing things, like how the Oakland
A's invented a whole new way of recruiting using math (which incidentally no
longer helps them because all the top teams adopted it and are back on top
again).

It's all about network effects.

~~~
cpsempek
Unsure of the exact metric you're referring to, and I'm about to be just as
informal, but sports teams swing much more wildly and often than universities.
In the NBA, most recently GSW, going from 5 straight finals with 3 wins to
last in western conference. See also CHI before and after MJ era. LAL most
recently after Kobe's retirement and pre-Lebron. In the NFL, SF post-Young
pre-Harbaugh. I agree, there are often dynasties and teams can sit in the tops
ranks for quite a while, however it is not at all uncommon to see once top
ranked teams vying for lottery draft picks. Maybe the over-all mix does not
churn that much, but I can't recall the last time a top 10 school dropped out
of the top 100 without a scandal involved.

~~~
jedberg
I would say sports just moves faster than academics. The average sports career
is only a few years with a few exceptions, the average academic career is much
longer.

~~~
ghaff
That's true and, furthermore, a big part of why a university is well-ranked in
year N boils down to it being well-ranked in year N-1 and N-2. With a sports
team, if you have some critical injuries or otherwise lose important personnel
from the prior year or two, no one cares that you won the championship a
couple years back if you have a mediocre record this year. There's just a lot
more inertia in university rankings both on objective factors and on things
that basically depend on being a good university because they're a good
university.

------
covidacct
I'm mostly on board with the critique of university rankings, but I really
don't buy this specific critique. It's a strawman.

Here's what they did: they took the top research universities and the top
liberal arts colleges. They then observed that the rankings can't distinguish
between the top institutions within each class, and that you can also flip
which of those two classes is preferred.

But, in what sense is it even meaningful to compare Amherst College and
Harvard [1]? Those are just enormously different types of institutions. It'd
be like creating a ranking of "best cars" that includes the top 5 sedans and
the top 5 trucks, and then observing that you can jiggle the rankings to get
anyone on top. Is a Toyota Corolla better than an F-150? IDK. Stupid question.
US News and World Report, for all its problems, _does_ at least get this much
right! they break down institutions by "type" and then rank within type.

Additionally, just because rankings are noisy and easy to game locally doesn't
mean they are inaccurate or easy to game globally. Two institutions within
10-20 slots of one another are probably pretty similar and rankings aren't
particularly helpful / are easy to game. But the #100 liberal arts college is
probably not as good as Amherst, and the #78 National University is probably
not as good as Harvard, and no amount of gaming is going to change that.

Rankings are indeed noisy and inaccurate and easy to game. But this particular
article is not a compelling demonstration of that fact.

[1] For non-US readers: Amherst college (not to be confused with UMass
Amherst) is in a class of peculiar institutions that are fairly unique to the
USA as described here:
[https://en.wikipedia.org/wiki/Liberal_arts_college](https://en.wikipedia.org/wiki/Liberal_arts_college)
TL;DR: they're basically the diametric opposite of super-charged research
universities like Harvard.

~~~
adas0693
Thank you for your feedback. I am one of the authors of this paper.

I don't think the paper gives any hints on us dividing universities into
classes and comparing. Would love to know how you reached that conclusion.

At the same time, yes, we could have divided universities into different
classes: research vs liberal arts, this vs. that state, big vs. small size,
etc. These are all trivial groupings but none of these would change the
conclusions of this paper. For all practical purposes, we could easily have
replaced the university names with labels like U1, U2, ... and still the
conclusions would not change. What matters is how a weight-based composite
index can be gamed and the paper does show that in multiple ways. Pls review
the ILP formulations yourself and run them on the dataset of your choice.

------
spideymans
Why is there such a fixation on university rankings in the United States?
Postsecondary rankings don't seem to matter as much in other Western
countries.

~~~
plorkyeran
_Is_ there actually a big fixation on university rankings? I had no idea what
my school's ranking was (apparently it's now #40) and I'm not sure I've ever
had conversation where university rankings came up outside of specifically
talking about where a high school student is thinking about applying. Even
then it's not like it's common for people to pick which school to go to out of
the ones that accepted them based solely on rankings.

People who care about Ivies and the such don't care _because_ the Ivies have a
high ranking. Someone who went to Harvard who wants to use that fact to their
advantage isn't going to say that they went to a "top 3" school.

~~~
AtlasBarfed
If you dropped 200k-300k on an education, you'll be invested in either the
continued value or and improvement in your investment's reputation, which as
the years go on is really its only value.

~~~
plorkyeran
200k is about what I spent on my education and I can't say I've really had any
reason to care about my school's reputation past my first job after
graduation.

~~~
AtlasBarfed
Tech jobs generally don't care, but traditional management and those tracks
really care about that.

Because getting a job in those areas is about abstract abilities that can't be
(somewhat) measured with code tests, technology signifiers, and tricky
questions like tech interviews.

So a certain school is a "class entry cue" for such tracks.

------
downerending
Regardless, you should think hard before accepting at the "top" university
that will take you. Many times it's better to be a big fish in a small pond
than vice versa.

Also, in terms of learning, it's really about the access you can get to the
ten or so percent of the faculty that actually worth anything. Figure out who
they are and go after them.

Also, network, network, network. Smart or not, everyone you meet in school
might give you a reference/job/funding someday.

(source: hard experience)

~~~
ordx
Care to elaborate? Did you have any problems after graduating from the "top"
university?

~~~
downerending
More "during" than "after". Not sure if mine counts as "top", but most would
say it's easily in the top 25 for my discipline. It was plenty hard for me,
though, and in hindsight, I think it would have been better to be the
brilliant kid in a lower-ranked school than an also-ran in a higher-ranked
school (relative to the range of schools that would admit me).

That's just a guess, but I've seen a lot of others suffer from shooting too
high.

------
notechback
University rankings have two key issues:

1\. They are significantly biased to English language (universities/research
output) and bigger countries in general 2\. They are totally irrelevant from
the student perspective as they mostly focus on research output, which means
that you'll get the richest universities and the best professors will be
writing papers rather than teaching bachelor level courses.

There are some attempts to fix these issues (see eg the EU "Multirank" which
allows you to choose your priorities), but it's still hit and miss.

~~~
jseliger
Those are useful issues, to be sure, but there is a lot more incoherence than
those two: [https://www.newyorker.com/magazine/2011/02/14/the-order-
of-t...](https://www.newyorker.com/magazine/2011/02/14/the-order-of-things)

------
finolex1
Gaming of rankings aside, the ratings organizations themselves offer various
'services' to colleges looking for a reputation boost.

[https://www.insidehighered.com/news/2013/05/29/methodology-q...](https://www.insidehighered.com/news/2013/05/29/methodology-
qs-rankings-comes-under-scrutiny)

For instance, QS sells a 'star-ranking' system for schools. Coincidentally the
schools who paid for a '5-star' ranking, such as the Unviersity of Bristol or
the Universiti Malaya are also placed higher on the actual QS ranking than
those that did not, including Georgia Tech, the University of Washington,
Ecole Polytechnique and UIUC.

~~~
alexpetralia
Related to expenses..

"And there are several perverse incentives in the marketplace that make it
hard for colleges to cut costs. The most basic one is that the U.S. News
algorithm rewards them for spending a lot of money: Higher faculty salaries
and more spending on student services lead directly to better rankings. If you
reduce your expenses, your ranking will fall, which means that next year your
applicant pool will probably shrink."

You'd be surprised at the burgeoning expenses of universities, and often
meager profitability, despite record-high tuition rates. It seems, often like
startups, a burn rate signals potential growth!

[https://www.nytimes.com/interactive/2019/09/10/magazine/coll...](https://www.nytimes.com/interactive/2019/09/10/magazine/college-
admissions-paul-tough.html)

~~~
mattkrause
I'm gonna go out on a limb and say that above-market faculty salaries are
_not_ a major issue.

~~~
alexpetralia
I would argue its headcount growth (not salary per employee), but admittedly I
could not immediately find data for this.

Edit: Also upon looking through the Trinity College financial statements, a
huge portion was categorized as "Other", which included consulting fees,
travel, interest expenses and various other expenses.

------
neilv
[https://www.bostonmagazine.com/news/2014/08/26/how-
northeast...](https://www.bostonmagazine.com/news/2014/08/26/how-northeastern-
gamed-the-college-rankings/)

> Freeland swept into Northeastern with a brand-new mantra: recalibrate the
> school to climb up the ranks. “There’s no question that the system invites
> gaming,” Freeland tells me. “We made a systematic effort to influence [the
> outcome].” He directed university researchers to break the U.S. News code
> and replicate its formulas.

~~~
augustt
Yeah this kind of gaming is probably widespread. UChicago is known for
"artificially" driving down their acceptance rate with lots of decision rounds
and marketing. Cornell isn't releasing their numbers at all this year - I'd
guess because they don't like them.

~~~
neilv
I've heard many allegations about established universities gaming their
ranking by, e.g., soliciting applications widely -- not so much to have a
larger and more diverse pool from which to choose -- but to maintain/lower
their acceptance rate metric.

Two things I like about the Northeastern story are that the official
demonstrated even more ambitious gaming, and it was publicized.

No criticism of Northeastern; they have many great people, doing great work.
And I've heard many faculty objected to the rankings-climbing emphasis as it
was happening.

------
skat20phys
Sometime not too long ago there was a blog post or article covered on HN,
about data showing student scores on some academic achievement measure over
time. If I remember, all students generally improved on this measure
throughout undergrad, and there was some average tendency for universities
with higher average entering standardized test scores to have higher scores,
but the effect seemed to be mostly driven by the students. That is, it wasn't
that some schools accelerated scores more, it was that they just tended to
select for higher-performing students. Also, the signal provided by school, if
I remember, was relatively weak. The author was basically arguing that
universities in general provide some educational value (or students just learn
over time; there was no control of students who didn't go to college) but that
the ranking was by and large just a signaling thing.

I can't find it now though because all the search terms I can think of just
return a bunch of other stuff related to universities and students.

~~~
heyn05tradamu5
Interesting. I wonder how one would measure whether an institution changes
student trajectories? Or if that data is available somewhere.

------
asaph
If you only read one sentence from this article, it should be this one:

> In this paper, we take a fresh look at this ranking scheme using the public
> College dataset; we both formally and experimentally show in multiple ways
> that this ranking scheme is not reliable and cannot be trusted as
> authoritative because it is too sensitive to weight changes and can easily
> be gamed.

------
tyingq
There is a very current thing happening where individual professors have too
much power. The whole "remote teaching" thing is VERY new for some
universities and professors. My son recently survived a final where > 90% of
the class failed. Mostly due to poor execution of remote learning. Including
simple stuff like classes overlapping meeting times, study material uploaded
too late, etc.

No pressure on the professor or university, though. All on the students.
There's a rant letter, from the prof, about effort where effort wasn't a
factor. My kid survived solely because he has a very good memory. No
accountability. I AM PISSED.

~~~
Aperocky
> where > 90% of the class failed.

I can't imagine this happening in a college class where the instructor get to
keep his class next year.

~~~
tyingq
U.S. State college, a fairly well respected one, happening now. Chemical
Engineering. They have no idea how to operate "online/remote". And that
matters for classes like Fluid Dynamics, Organic Chem, Differential Analysis,
etc. Is there some realistic path for me to escalate?

~~~
covidacct
_> Is there some realistic path for me to escalate?_

Is the instructor an ad junct? Those are cheap and replaceable. They make
maybe ~1/10th of what your average FAANG engineer makes because the labor
supply is extraordinarily over-saturated. Enough complaints will probably get
them fired, if that's what you're looking for, but realize you're probably
putting someone with no prospects and no savings out on the street and they
will almost certainly be replaced with someone who gives about as many shits
as you can expect from someone with a PhD who makes 30k a year. That tuition
you're paying isn't flowing to the people teaching the courses.

If the instructor is an actual professor, then probably there's nothing you
can do.

BTW, let your kid at least try their hand at navigating this on their own,
even if you provide oversight. Barely competent people doing a poor job is not
unique to pandemic times and is definitely not unique to higher ed. Knowing
how to use a bureaucracy to route around an incompetent person not doing their
job well is a valuable life skill. Plus, everyone involved is going to be more
sympathetic to an articulate student than to a "helicopter parent".

~~~
tyingq
Not a big fan of the "helicopter parent" term. I'm paying for it, so I want to
understand what I'm paying for. Why is that a "helicopter"?

------
dontbenebby
I think they're useful-ish, but it's important not to get too focused on
specifics.

For example I did a stint in policy and I'm reminded of how everyone insisted
for a long time that "top" law schools are the "top 14" or T14 for short. This
is historically the measure used since Georgetown was in DC, and the children
of the elite must be "top tier" regardless of if their school ranked outside
top ten on every official metric such as incoming LSATs, publications, etc :D

~~~
barry-cotter
That’s not where T14 comes from.

> There exists an informal category known as the "Top Fourteen" or "T14,"
> which refers to the fourteen institutions that regularly claim the top spots
> in the yearly U.S. News & World Report ranking of American law schools.[8]
> Furthermore, only these fourteen schools have ever placed within the top ten
> spots in those rankings.[9] Although "T14" is not a designation used by U.S
> News itself, the term is "widely known in the legal community."[10] While
> these schools have seen their position within the top fourteen spots shift
> frequently, they have generally not placed outside of the top fourteen spots
> since the inception of the rankings.[11] There have been rare exceptions to
> this, however, such as UCLA School of Law appearing in the top fourteen
> instead of Cornell and Northwestern in 1987 and University of Texas School
> of Law displacing Georgetown in 2018, although the significance of these
> changes has been debated.

[https://en.wikipedia.org/wiki/Law_school_rankings_in_the_Uni...](https://en.wikipedia.org/wiki/Law_school_rankings_in_the_United_States)

~~~
dontbenebby
>> That’s not where T14 comes from.

>There have been rare exceptions to this, however, such as UCLA School of Law
appearing in the top fourteen instead of Cornell and Northwestern in 1987 and
University of Texas School of Law displacing Georgetown in 2018, although the
significance of these changes has been debated.

People select measures for a number of reasons. It's perfectly possible the
single and only reason T14 is used is for the reasons you lay out.

OTOH I have personally had conversations with people who argued that the T14
is a measure of merit when Georgetown was in the T14 then " _debated the
significance_ " of the measure when Georgetown dropped out of the top 14 in
2018.

It was not clever or endearing.

------
jdmg94
I personally think Universities and colleges are a scam that was necessary
before we had access to the bulk of human knowledge on the tip of our fingers.
We should go back to an apprentice system, go through elementary and middle
school and then find a profession you would want to learn OTJ, this would
shift the responsibility of training to companies but the people coming out of
there would be immediately productive.

------
tygrak
Rankings are a bit dumb, especially because any non english university is
going to get ranked terribly basically on every of these rankings. There's no
way that the oldest university in central Europe, started in 1348 should be
placed 400-500th. That's just a joke.

------
MR4D
The only ranking that matters to the vast majority of people:

Median starting salary / median cost to attend.

Sure, there are other considerations such as. If it will get you into a better
grad school or something, but for most it’s really this simple.

------
eindiran
Here is the code used by the paper: [https://github.com/alidasdan/university-
rankings](https://github.com/alidasdan/university-rankings)

------
thinkloop
Any idea what they are measuring against? For example avg. income or accolades
of graduates.

