
We Need a Yelp for Doctoral Programs - etattva
https://community.chronicle.com/news/2283-why-we-need-a-yelp-for-doctoral-programs
======
ISL
Good luck getting honest reviews. Graduates and recent post-docs are dependent
upon good letters of recommendation from their advisors for perhaps a decade
after leaving their research groups.

The truly disillusioned won't mind complaining, but those with even
constructive criticism will feel constrained in their ability to speak out and
be specific.

The graduate-student experience is deeply advisor-specific. Professor A may be
exploitative, while Professor B may fight tooth-and-nail for student success.
The statistics are low, too. Most professors will only graduate a few students
per decade.

~~~
jseliger
In many disciplines, too, the answer is "don't go to _any_ program." Most
humanities fields, for example, have numerous essays written about why doing
grad school in them is a poor life choice, like my contribution:
[https://jakeseliger.com/2012/05/22/what-you-should-know-
befo...](https://jakeseliger.com/2012/05/22/what-you-should-know-before-you-
start-grad-school-in-english-literature-the-economic-financial-and-
opportunity-costs/).

One could say the same of most social sciences.

~~~
tharne
I was at a social event and got to talking with a professor at an Ivy League
school. He had a really insightful take on this. He said that the only fair
way to present a doctoral program to prospective students is to compare it to
trying to become a rockstar.

Every day tons of people form garage bands and have dreams of becoming rock
stars. But...they do so knowing they almost certainly won't make it.

This professor said that starting a doctoral program should be viewed the same
way, as a sort of moon-shoot that will take a lot of time and resources from
other pursuits. If you're okay with that, go for it, and if you're not, find a
path that works better for you.

Unfortunately, the administration of the school he worked for told him he
would be removed from his position if he shared this philosophy with
prospective students.

~~~
scott_s
That is a good framing. When I explain it to people, I compare it to the NBA.
Except it's like trying to get into the NBA when all of your coaches over your
entire basketball life up until that point have _been_ in the NBA. Proximity
can trick you into thinking you just have to "decide" to pursue such a
position, not knowing that such positions are the result of a ruthless
tournament system.

~~~
xhkkffbf
The thing is that there are more prosport athletes hired each year than there
are new professor slots in most disciplines. There are probably 3-5 new
players on each NBA team each year which amounts to 75-150 people taken from
the draft. Most academic disciplines have only a few dozen slots available
each year.

~~~
scott_s
The reason I pick the NBA is because there are only 450 positions available
(30 teams, 15 players per team). This is the least number of total positions
in popular team sports in the US. The ratios will never be perfect, they just
need to be relatively similar. The point is to use something which people
already have an intuition for; everyone already has an intuition for how
unlikely it is for any given high school or college basketball player to play
in the NBA.

------
supernova87a
The problem here is that a Yelp rating system for restaurants can be consumed
by anyone because everyone wants a good-tasting, cheap, pleasantly-served
meal. There's not a lot of range of desire and recipient goals.

Grad students on the other hand, come with a huge range of:

\-- Motivations for doing a doctorate (say, purely for job prep, versus
wanting a life of academia and intellectual fulfillment),

\-- Skills and knowledge (some already essentially begun their doctoral work
and have achievements already, versus those who will be wandering around
searching for a topic for years),

\-- Tolerance or willingness to engage in ambiguous, possibly dead-end leading
places with no clear sign of success,

\-- Need for financial reward / subsistence and productive return for time or
opportunity cost traded off.

Add to that in many fields, the supervising professor is the greatest
variable, more than the program or university -- and any rating system will be
irrelevant or worse, misguiding. One person's paradise could be another's
hell. Unless the rating is so dumbed down to generic factors in which case
what are you really informing people about?

To echo some advice of a contrarian grad school dean, go into grad school
because you know what you want to write about, who you want to work with, and
what you want to come out the other end with. Otherwise, the bulk majority of
people going into grad school "on autopilot" or based on a Yelp rating are
going to be sorely disappointed at some point.

------
Upvoter33
The problem in doing this is that saying "school X is great for getting a
Ph.D." will never likely generally be true, because the Ph.D. process is so
advisor dependent. However, doing this for each advisor is really hard too,
because the sample size is small and it's hard for the reviewer to remain
anonymous.

All of that said, some set of general statistics (size of program,
distribution of research group sizes, percentage of those who wanted a Ph.D.
and then got one, etc.) plus a general satisfaction survey among graduate
students might be a good complement to the usual ratings one sees in US News.

~~~
refurb
Underrated comment. The “school” rating is meaningless. Even the “department”
is meaningless. It all comes down to your advisor.

Our department had amazing advisors and horrible advisors. Two people could be
in the same PhD program and have polar opposite experiences.

In fact, if you want reviews it’s not hard - just ask current students about
_other_ advisors. Rumors move fast and I’ve found you’ll get an honest
opinion.

------
abakker
We barely need a Yelp for restaurants. I think it is time for the internet to
acknowledge that the pointless drivel in star rating systems is just not
helping anyone. If you’re going to start a system for rating things, give it
more nuance and value.

~~~
fastball
I actually have a plan for this!

I want to build a ratings website where you rate things in the same way they
generally structure psychological tests, by responding to affirmative
statements with something between "strongly agree" and "strongly disagree".

As an example, if you were rating films, questions could be:

"This film was enjoyable."

"This film had an engaging plot."

"The acting was believable / appropriate for the film."

etc. Ideally you'd have about 5 standard statements that you respond to for a
particular thing you want to rate (whether its films, books, food, or a
university course).

Better than a star rating system, because the problem with stars is that you
end up with scores that always hover from 5-8 (if out of 10) in a very not
useful way (see IMDB).

It's also better than what RT / Netflix / many others have moved to (just
voting with up/down) as those require _a lot_ of data in order to really give
interesting results, where as with this system you can start to find
interesting insights almost immediately, or within a single person's own
collection.

~~~
abakker
I like this - in general, I’m a fan of this (my job is survey research). There
are many different approaches to getting better results than most reviews get.
Personally, I’d love to see a max-diff approach to reviews.

~~~
fastball
Feel free to steal the idea! I'm a bit busy building out another idea I had
haha.

------
eli_gottlieb
No. We don't need a consumer review website, because PhD students (I am one)
are not the _consumers_ of doctoral programs. We're the workers. We need a
union.

If we're in a _good_ program, we're receiving valuable training both in our
coursework (or before PhD in an MS/MA if your country does it that way) and in
our research work. Ideally, we're building up a network of collaborations and
acquaintances in the scientific community that gives us an "in" for the next
stage of our career, while also being mentored by world-class experts in our
fields.

In a not so good program, we're used as TAs for a few years and then rapidly
encouraged to drop out.

Either way, most of what we do is not like most of what an undergraduate
student does. It is like what an entry-level professional does.

~~~
chrisseaton
On a PhD you work for yourself - you shouldn’t need a union against yourself!

~~~
eli_gottlieb
>On a PhD you work for yourself

What field do you work in? Modern science is team science. I think the paper I
have under review right now has... six authors? That's pretty average for the
particular interdisciplinary collaboration I happen to work in right now.

It would just be _silly_ to claim that every other junior author on the paper
is working _for me_ , as the first author, rather than admit that I work _for
my adviser_ (who I haven't complained about here because I lucked out and got
a very good one).

~~~
chrisseaton
> What field do you work in?

Computer science.

> It would just be silly to claim that every other junior author on the paper
> is working for me

Exactly - the all work for themselves. I'm sure they all have their own
research goals and they come together on papers along the way.

But what you need to focus on is the publications you achieve. You achieve
them for yourself not your advisor.

------
catsarebetter
Doctoral programs are at least 4-5 years long and crazy emotionally
challenging as well as a pretty surprising drop out rate, if someone doesn't
take time to research them deeply before they commit to something like that
then I think that person needs to learn the lesson that that will teach them.
Yelp is the surface level easy way. Networking and cold-calling tons of
current phd and master's students over the course of undergrad and during
internships is how everyone does it.

------
vnorilo
I am not sure a Yelp-like system would work for the reasons others have
mentioned.

I would suggest these metrics to help you evaluate a doc program from the
outside:

Do early-stage grad students get (any) authorship credits together with
professors or post docs? (Getting started)

Do they eventually get first author status? (Independence)

Do students go to conferences and read papers? Or does a prof just go and read
all 10 from her institution? (Networking)

Do fresh doctors land post docs, tenure, or good private sector jobs?
(Outcome)

A lot of this can be deduced by google (scholar) and scanning conference
proceedings and video recordings.

(defended my doc in 2016 in Finland, been fairly succesful at post doc funding
as well as private sector)

~~~
kayaeb
protip: if your prof is tenured and still first author on more than 2 papers a
year, they're siphoning their student's work, or they're setting you up for
pipelining positions (which are not horrible, but, be aware that's what's up).

I've never had a prof insert my work into their own papers, they've hooked me
up to contribute to other teams, but with us it was always "my" project.

If you get a chance to see them present at a conference (lots of these are on
video online nowadays), check if they specifically mention their students in
the presentation, that's a green flag.

~~~
vnorilo
Good observation.

Writing as second author was a very positive factor very early on. I was an
assistant to a research fellow from my second year at uni, mostly because of
coding skills. Pretty soon they encouraged me to write a pragraph or two about
the implementation details, and I got added to the author list. I felt that
was appropriate and a great kickstart to a research career (got my first MIT
press journal credit before my bachelor)

I agree however that professors growth-hacking their publication lists with
student labor is problematic indeed and looking at first/second author creds
is a good way to flag such behavior.

------
noelwelsh
Any time education is discussed on HN it should come with a mandatory notice:

\- Your experience is no necessarily indicative of others' experiences

\- The US education system is not indicative of education systems in the rest
of the world.

------
ramraj07
We dont need a Yelp for doctoral programs, we need a Yelp for research labs!
It's more about the lab you join than the program itself that makes or breaks
your life.

In fact, I have heard that minimally there's kind of a toxic labs blacklist
maintained in some Chinese message boards. At least that's what a fellow
postdoc told me when I mused why we suddenly stopped getting applicants from
China to our lab!

~~~
frandroid
There are PhD programs not attached to research labs. I would say the majority
of doctoral students are not attached to one.

~~~
mmhsieh
by research lab, he means a research group led by a specific faculty member in
an academic department. in other words: he is talking about reviews on
specific professors as doctoral degree advisers.

------
michaelmrose
So someone to extort colleges to remove bad reviews?

~~~
etattva
Ha ha I know where you are going :) My take from the post was that there needs
to be some information provided to end users based on which they should be
able to decide what colleges to choose. Ranking thatis dictated by third
parties cannot be a criteria. And student should not spend too much time
emalining each and every college they are interested in.

------
dhosek
I've done my best to create a ranking of graduate creative writing programs by
measuring the appearance of programs' graduates in the top prize anthologies.
It's been generally well received although there has been some resistance to
the idea of ranking programs like this. There's still a lot of work to be done
on it, but the basic structure, at least is in place.

[http://creativewritingmfa.info/rankings/](http://creativewritingmfa.info/rankings/)

~~~
etattva
nice work. How do you keep it udpated?

~~~
dhosek
There's a fair amount of manual data collection. I get an excuse to enter into
e-mail correspondence with 100s of writers each year, although I have most of
the initial contact programmatically driven. There's also automatic annual
e-mails of program directors in August to get the latest data from them and
for the top schools on the list, I'll make calls if they don't answer the
e-mails. In the fall I end up spending an hour or so a day dealing with all
the e-mails and data entry.

------
devalgo
Just want to point out that many of the complaints in the comments here are US
specific. At many PhD programs in Europe for example you are considered an
employee of the university and not a student, this often comes with full
benefits: salary, bonuses, vacation time, pension, etc. Academia in the US in
particular is falling into disrepair.

------
brodo
As someone who dropped out of a PhD program: the most important metric is the
drop out rate for the research group you are in. If you can get it for your
thesis advisor, it's even better. There are people who see all graduate
students just as cheep, highly motivated labor for boosting their academic
output. I had to put my advisor on papers who he had never read and I know
someone who had to "ghost write" for her professor. Academia is a status game,
and giving someone else credit for your work won't get you anywhere.

------
westurner
How are the data needs for such a doctoral and post-doctoral evaluation
program different from the data needs for
[https://collegescorecard.ed.gov](https://collegescorecard.ed.gov) ?

Data:
[https://collegescorecard.ed.gov/data/](https://collegescorecard.ed.gov/data/)

Data documentation:
[https://collegescorecard.ed.gov/data/documentation/](https://collegescorecard.ed.gov/data/documentation/)

~~~
sgillen
Reviews seem like the biggest need.

“The culture in the X department is terrible, everyone works 80 hours a week
and is miserable”

“Night life is great at least, the grad students usually go out for student
nights at blah blah after seminar”

“Underrated X program because at university Y, don’t be fooled we literally
have 4 of the 5 top researchers in field Z”

Etc.

~~~
etattva
But it is so hard for people to give reviews. People seem to give good review
when are they are very pissed off or very happy. Later seems to be a very rare
case.

~~~
p1esk
I couldn’t make sense of your comment.

~~~
liammonahan
I think they mean “good” in the sense of being helpful reviews: positive or
negative.

------
glial
It would be better to just contact current students at the lab and offer them
coffee for a conversation.

~~~
CarbyAu
While I agree with the sentiment, many potential students are looking
overseas.

Is there an Internet equivalent of "coffee and conversation"? Steam game gift
and discord maybe...

~~~
henrikeh
You don’t have to bribe them... you can just write to them.

~~~
CarbyAu
A lot of people will say things they won't write, especially in a nice
environment.

~~~
henrikeh
The schedule a call when you found someone you can talk to.

Of course you lose something when physical presence is impossible, but in my
experience you can always find someone who is happy to write, chat or call.

------
aspaceman
This article really rubs me the wrong way. I don't think I would find a Yelp
very valuable, and the most important metric for a PhD program isn't job
placement numbers. The idea that job placement correlates to PhD quality is
just plain wrong. Incredibly intelligent students leave PhD programs through
no fault of their own, and people who get nothing out of a PhD program can get
a cushy job easily. It's a horrible metric.

I found that the difference between faculty members at the same institution
can be immense. This is because faculty have great control over your life due
to how doctoral funding works in the United States. If you wish to continue a
Biology degree, you must be willing to work with this faculty member, in this
area, on this specific problem. Some faculty are rigid about a specific
problem, some are not.

Even more complicated is that some students need more guidance than others. Or
some students want much less. And the student doesn't necessarily know what
they need before entering the program.

This is why I think it's smarter to give doctoral students more freedom and
flexibility in switching faculty. The school should be supportive of a student
in this process rather than focusing on their job placement numbers.

~~~
foobarian
I agree with you, and further think it would be awesome if each doctoral
program offered "user manuals" for each potential advisor in the department.
Things like, "Professor X is very hands off, so you'll be able to do a lot on
your own, but make sure you get on his calendar to get advice." Or, "Professor
Y is a slave-driving task master but very capable; so make sure to stand up
for yourself or you'll get crushed." Of course this is tongue in cheek and
would probably never get formalized.

I got this kind of advice from people who already finished their PhDs while I
spent a year in industry before grad school. Not everyone can do this but
maybe there is some other way to cold-contact finished PhDs and see what they
know about the faculty you're considering.

~~~
mmhsieh
there are ways to do this but the market for the manuals would be pretty
small.

------
andrewacove
Even if you have a terrible experience in your doctoral program, much of its
outward value is in the program's reputation. Giving a program a negative
review hurts the perceived value of your own education.

------
dhosek
Basic rule: if you're paying for a doctorate (aside from a professional
degree, like an MD or JD), it's a bad idea.

------
r-w
I think Glassdoor[1] would be a better analogue.

[1] [https://www.glassdoor.com/about-us/](https://www.glassdoor.com/about-us/)

------
croissants
If you're debating whether to go to school x and work with professor y, make
sure you get a chance to talk to students who work with professor y, and
ideally students who work adjacent to professor y. In my experience, bad
advisors usually have students who will tell you that the advisor is bad
(especially if you talk to them in person or over the phone, where there's no
written record). If the students themselves are too scared, their friends
probably aren't.

Any good program will have a built in process, for example an admitted
students event, with this kind of interaction. Any good advisor will encourage
you to talk to their students. If not, that is already a red flag.

Talk to current and former students! I was fortunate to have a great advisor,
and I was always happy to talk to admitted students.

------
physonaught
A good mental model for graduate school is that it is a feudal system. There
is essentially no oversight of professors by a larger executive branch. I
mean, they can't literally murder you, but they can drive you to suicide.
(Jason Altom)

If you get lucky, you get a professor that believes in you (this can be one of
the most rewarding experiences I know of).

If you aren't lucky, then things will get really rough. I've seen professors:
\- Spike their student's candidicy at the last minute (meaning the student had
to switch groups in their last year, essentially having to redo their PhD in
another group) \- Spike student papers \- finding out where their former
student was applying for jobs, calling and spiking their job application
(talking shit) \- various abuses (yelling at student for not working on
Christmas, humiliating student for years in public, sexual harassment)

I could go on. If you haven't been through grad school, you might ask "why not
leave/vote with your feet?"

But understand the power dynamic: in really broad strokes, the student works
towards a reward (being a Dr/PhD) that will take, on average, 5-6 years.
Getting that reward is completely up to your supervisor's discretion. It's
like if you worked at a startup for six years for minimum wage, but if your
boss agrees to it, you will get a million dollar payout (also, your boss can't
be fired, ever.) And that's just a minimum-- success in academia is dependent
on getting full throated letters of rec from your supervisor, so they also
have to really like you.

So your professional success is completely up to one person: if they decide to
fuck you over, that can be 4 years of work down the drain.

When I was in graduate school, I never saw the faculty/department intervene in
any of the abusive situations. The only thing that protects students are soft
processes, like knowing which professors are really bad. The Yelp idea is
good, simply because it increases that ability.

And those abuses I talked about? A majority of my cohort dealt with that (at a
large, well known University), it's not isolated to a few bad apples.

------
crb002
The last time I tried to interact with Iowa State University ECprE, a
professor who I won't name asked me for a $20,000 bribe to do research with
their lab. Confirmed their culture is just as toxic as it was a decade ago.

~~~
ausbah
why not nane and shame?

------
euix
A PhD is a monastic experience, I remember reading that on a blog somewhere.
Couldn't agree more.

It comes with all the same features (it's even baked into the history of
education being baked in to the Church).

------
bosswipe
There's a simple objective metric that could capture most of what you'd want
to know about any school and put USNews's rankings out of business: graduates'
achievement. How much are graduates making 10 years later? How many have
prestigious jobs or other accomplishments such as research grants or awards?
Universities do track these kinds of metrics as part of their alumni
fundraising efforts but they're rarely publicly disclosed.

~~~
michaelt
_> most of what you'd want to know about any school [...] How much are
graduates making 10 years later?_

Among my friends in graduate school:

* One was told by their adviser they had enough for a thesis in their second year, and found time to organise a university-wide Dance Dance Revolution championship.

* One was shouted at by their adviser for not coming in to tend to their experiments on Christmas day - the first day they'd taken off that year.

I think a prospective student would like to know which of those experiences
they're in store for, regardless of future earnings.

Indeed, if you want to maximise your salary, I'm not sure you should go to
graduate school at all...

------
aj7
The variation among thesis advisors within a given doctoral program is
sufficiently great that it should be

We need a Yelp for doctoral thesis advisors.

I think the author was reluctant to “go there.”

------
nabaraz
A decent review site would be awesome.

I have been looking at WGU for an online MBA degree. Reviews are just
scattered everywhere from Reddit to affiliates websites.

~~~
bruceb
Can I ask why WGU? Instead of at least a mid-tier school ?

~~~
OldHand2018
WGU is accredited. If you are sufficiently motivated, it is fast and cheap. It
doesn't require you to move or to quit your existing job.

I can't speak to the motivations of the person considering it, but perhaps
s/he only needs to check a "do you have an MBA?" box on an application to be
able to move forward in their career.

------
trombonechamp
There was a news article about this in Science two years ago, which a couple
such efforts: [https://www.sciencemag.org/careers/2018/02/crowdsourcing-
goe...](https://www.sciencemag.org/careers/2018/02/crowdsourcing-goes-
academic-platforms-reviewing-advisers)

------
etattva
Forget about yelp and ranking and rating. Ithink the folks realize there is
not a single US website that lists all the graduate schools and the courses
details. We are talking about solving the next step of the problem when basic
aggregation is missing.

------
hymnsfm
Not a fan of Yelp, but I see a need for a Yelp-like lookup for landlords and
superintendents.

------
tomashertus
I was thinking the other day that I need a Yelp-like experience for technical
tutorials and creators of educational content online.

I'm sure I'm not the one who ran into so many shitty tutorials/how-to
guides/videos through the years.

------
ganfortran
For real.

But the title itself is something worth a moment of reflection.

Doctoral programs are supposed to be scarce, selective and of high profile.
Now they are being treated as a service that can be requested and compared
against.

~~~
kayaeb
This really rubbed me the wrong way, the closing sentence sums it up: "What if
a doctoral program’s prestige arose, in part, from the way that it treated its
students? We should dare to dream of such a thing."

There is a huge problem in academia where the number of PhDs is growing
exponentially, and the number of positions is staying basically flat. I
strongly disagree with this "participation award" mentality of "you deserve
it." It's... leading a lot of people to make really stupid career decisions
and I think a lot of people are doing it just to get this gilded club to
bludgeon people who don't have PhDs -- that's the only reason I can think of
for a person to hope for an "easy" PhD experience.

My advisor being rough at the right times (and in good spirit) was absolutely
essential to the defense, which is essentially gladiatorial combat against
your father (well, mother in my case), when both you and your advisor hope you
can defeat the master in your specific realm. I would not have been able to do
it if my advisor pranced about making everything pleasant and easy and
convenient.

------
blueboo
The problem isn’t that we’ve locked opinion inside communities. It’s that
doctoral programs are so poorly run. That universities are waiting for
quantitative data to instill action is more sign of dysfunction.

Here’s yelp for grad programs. Write down the name of the program you’re
looking at and make an asterisk. One star — it’s already more accurate than
Yelp on restaurants.

------
etattva
collegescorecard measures the outcome. In the post, Profesor Leonard talks
about doctoral programs being rated based on research outcomes.

------
unemphysbro
ejmr exists

