
The Scent of Bad Psychology - coldtea
https://putanumonit.com/2018/09/07/the-scent-of-bad-psychology/
======
lellotope
It's been said before but needs to be said again: this is happening everywhere
in the biomedical and related fields. The neurosciences, oncology... the list
goes on and on. Anyone take a look at AI research lately? How much tweaking is
going on there? How much is your big data finding due to the idiosyncracies of
your particular dataset?

Psychology, as has historically always been the case--meta-analysis itself
bloomed largely from the field--is the one turning inward and looking at
itself. And it's getting crap from people who love to use it as their favorite
punching bag.

The irony of this article is that it's psychologists looking at other
psychologists, doing the math etc.

The truth is closer to the second hypothesis by the author: bullshit is
incentivized everywhere in academics. Reality is less interesting, harder,
more incremental. Everyone wants the next genius savior to point to because
it's a simpler story than reality. Sexy means more pubs, more grant money.

~~~
m_mueller
I’ve recently been at a master students’ thesis defense where a method was
discussed that is supposed to speed up algorithms by deciding whether to run
them on GPU or CPU. The method is to look at the program code with deep
learning algorithms.

Fair enough. There’s apparently a set of benchmarks that other papers used and
were reused here to compare against. Yet the student was unable to answer my
question as to how bias in this dataset was treated or even what percentage of
the algorithms was faster on GPU to begin with, they only showed how much
better their deep learning fit the data. As in, if 99% is faster on GPU I
could as well just provide that as a static answer and be better than any
machine learning.

I frankly find it tragic that this kind of stuff is not ironed out in
academia. This was was in a very highly ranked university.

------
Arun2009
The key takeaways for me from the 80000 hours test
([https://80000hours.org/psychology-replication-
quiz/](https://80000hours.org/psychology-replication-quiz/)):

> Papers that report boring or common-sense results seem to be true very
> often. Cognitive psychology (e.g. memory and perception) mostly held up.

> ...it turns out for most people "if the claim of the paper doesn't make
> intuitive sense, the paper is wrong" is a good shortcut that works about 70%
> of the time.

Generally, the closer to the "metal" (e.g., memory, perception, learning) and
the more pedestrian the psychological finding is, the likelier it is to hold
good.

I might be giving in to a bit of hindsight bias here, but I have come across
this in the psychology of learning myself. It has been a bit underwhelming I
must admit. "Good sleep, retrieval practice, elaboration, paying attention,
meaningfulness and organization, and spaced repetition result in better
learning; massed-practice is not very useful for long-term retention". Well,
no kidding!

------
charlieflowers
Taleb's grandma seems pretty down to earth.

~~~
weberc2
Taleb himself is pretty awesome.

------
swirepe
Is priming really bullshit? I watch batters go through all sorts of little
rituals to get their heads in the game.

If you wash 5 times a day before prayers, yeah, washing your hands is going to
put you in a mindset.

Or maybe I just don't understand what priming is.

~~~
derefr
There was a recent finding (not yet replicated!) that performing rituals of
_any_ kind aids in building/retaining self-control/willpower.

This is kind of _like_ priming, but—if it holds true—this result would
actually take away a lot of the explanatory power of most priming studies. A
simpler explanation than a bunch of specific "doing X makes you think about Y
which makes you do Z" priming relations, is "doing {any number of things}
releases striatal dopamine, that increases executive functioning, which allows
you to do {any number of things}."

------
keyle
Is there a list of websites that discuss the latest research? Open to all
fields...

~~~
forapurpose
Nature and Science are excellent starting points.

------
o_____________o
> Imagine yourself washing your hands. Do you feel any impact whatsoever on
> your desire to rationalize decisions? Now imagine explaining this study to
> Nassim Taleb’s grandma.

What's the joke with invoking Nassim here? That he is some kind of anti-
pretentious truth-teller whose grandma is even further on that spectrum?

~~~
itsdrewmiller
The image at the top of that section is a tweet of him describing his "grandma
test".

------
forapurpose
The author offers rules of thumb for evaluating research, but is a business
school grad (MBA? BBA?) and "a data science amateur, from the Latin _amare_ –
to love." Am I misunderstanding something? On what do they base these
instincts and intuitions for evaluating research papers? If they wrote a blog
post on rules of thumb for evaluating code, would you take it seriously?

The "scent" here may be something on my shoe ...

~~~
girvo
[https://osf.io/8pc9x/](https://osf.io/8pc9x/)

As per the large amount of links in the article, the author seems to have
passed the test themselves, then talked about what heuristics they used to get
their result (that matched the studies linked). Did you read the article?

One can argue with their heuristics, of course, but dismissing them because
they're open about their background is a classic middlebrow dismissal. This
isn't presented as rigorous science.

~~~
forapurpose
> Did you read the article?

> classic middlebrow

That is classic unwelcome behavior on HN. Maybe you can keep your judgements
of other commenters and hot take reactions to yourself and stick to the
subject.

I didn't criticize them for being open about their background, but because
they lack expertise to offer expert opinions. Again, if they wrote an article
on rules of thumb for evaluating code or on reading radiology images for
cancer - things they appear equally expert in - it wouldn't be taken seriously
here.

> As per the large amount of links in the article, the author seems to have
> passed the test themselves

Huh? Lots of links pass some test? Reading a bunch of links make you an
expert? I guess the Internet really is full of experts.

~~~
coldtea
> _That is classic unwelcome behavior on HN._

I'd say middlebrow dismissal is even more classic:
[https://news.ycombinator.com/item?id=4693920](https://news.ycombinator.com/item?id=4693920)

> _I didn 't criticize them for being open about their background, but because
> they lack expertise to offer expert opinions._

You didn't criticize the author's opinions or any specific signs of
insufficient expertise however. Just their credentials.

Credentials != expertise (one can be an expert by self-studying, in fact in CS
many are).

~~~
forapurpose
What would be productive is discussing the expertise of the author, or if you
don't think that's of interest, not discussing it at all. I notice you posted
the original article; I'm not criticizing you for posting it but perhaps this
is getting personal?

Insulting commenters with derogatory names is unwelcome and lowers the quality
of discussion; I don't care if you have a link to it. Tangentially, it is a
good example of how even the smallest behavior by a leader, such as pg, set a
norm for many followers: people are digging up his posts 6 years later like
they are scripture - wisdom, law and conduct to study. pg is just a person,
not a prophet; YC and HN are wonderful, but his every act is not wisdom or
law. (No insult to pg; the same could be said of Linus Torvalds, Steve Jobs,
and everyone else who ever worked in SV or lived on Earth, including me.)

> Credentials != expertise (one can be an expert by self-studying, in fact in
> CS many are)

Credentials are a strong, widely used sign of expertise. The CS experts who
lack educational credentials are practitioners, professionals in the field;
they have other credentials. The author is neither educated in the field nor
is a professional or practitioner in it. Everyone else weighs those things
heavily. Do you hire or take the advice of people with neither education nor
experience in the domain?

The article in particular requires expertise: It's a series of rules of thumb,
things that are gained only through extensive experience doing the task
described. And it requires doing it with success - not all experience is good
experience - and apply good analysis to that experience. Rules of thumb for
physics research from someone who washed out are not the same as those from
Stephen Hawking; how about rules of thumb from someone who never has studied
nor practiced physics, but read some links? Again, if they wrote 'rules of
thumb for evaluating code' on the same basis, it would be ridiculous.

In general, everyone knows that the Internet is full of people spewing
nonsense they know nothing about, but even smart people often seem to fail to
apply that to particular instances, especially when other psychological drives
apply - such as when what is said moves or inflames them. The downvotes and
criticisms (this is an observation, not a complaint) seem to be because my
comment disrupts that drive (in this case, the inflammatory narrative about
social sciences). That's ok and what I expected. My prediction is that if I
wrote the same about a different topic, then the response would be different:
If what I wrote agreed with the crowd (e.g., if the author said psychological
research was high quality and I questioned their expertise), then I'd expect
many to agree with me, but for the same bad reason. Inflammatory stuff is,
disappointingly, still hard to talk about rationally.

~~~
coldtea
> _I notice you posted the original article; I 'm not criticizing you for
> posting it but perhaps this is getting personal?_

No, but the above is an attempt of second-guessing my psychology (and also
irrelevant).

Now, you follow with a long list of opinions of the importance of credentials.
What I don't see is any opinion on why the particular arguments are not
enough. Can you enlighten us? You could point to someone with credentials
which refutes them for example.

If you just want to raise a general principle (totally unrelated to the
quality or not of the arguments in TFA) then it seems that you suggest that we
should only discuss articles written by people with credentials on a topic.

If so, why should we accept this suggestion of yours then?

Where are your credentials on epistemology (seeing that this is an
epistemological issue)?

~~~
forapurpose
> a long list of opinions of the importance of credentials

Those haven't been addressed. I think calling them "opinions" is a big
stretch; it's a reliable statement of fact that humanity very often relies on
the credentials of the speaker, including their training and experience, for
evaluating information they communicate.

> What I don't see is any opinion on why the particular arguments are not
> enough.

It's up to the person making the claims to establish them. We can't spend our
time refuting everything anyone says. In fact, making claims in order to force
others to refute them is a well-known propaganda technique for creating
dysfunction, in part because refuting takes far more time than just making
claims. (This article is not propaganda, in case anyone thinks I'm implying
that.)

The arguments made by the author are expert judgments; there's little to
address by other means.

> it seems that you suggest that we should only discuss articles written by
> people with credentials on a topic

Personally, I want to spend my time on the best knowledge I can find. Life is
short. Identifying quality information depends frequently on credentials; I
think that's uncontroversial.

You keep taking things to an extreme - "only" \- and to be clear, those are
your words and not mine. Somewhere else I also made that explicit. EDIT: Here
it is, copied from another subthread to keep everything in one place:

 _Of course, let 's not make this binary, one or the other [merits or
credentials]. Everyone uses both to varying degrees, and other signals too (if
the post used bad English, for example, many would doubt the author's
credibility)._

 _In this case, I lean more heavily on expertise because the content is a
product of expertise - rules of thumb. If they wrote 'I can create cold fusion
in a teacup', I'd lean heavily on the merits - can they do it or not?_

