
“Default Position Should Be Skepticism” and other advice for data journalists - danso
https://www.propublica.org/nerds/hadley-wickham-your-default-position-should-be-skepticism-and-other-advice-for-data-journalists
======
distant_hat
The problem is incentives. As long as they are incentivised to maximize click
through rates, you'll get hyped up headlines. There really isn't that much
exciting stuff happening every day in science, and even when there is, someone
writing a clickbait headline will be better off than someone writing a
measured one. An excellent Twitter handle in @justsaysinmice.
[https://twitter.com/justsaysinmice](https://twitter.com/justsaysinmice) All
it does is says 'in mice' for all studies where hyped up headline and
sometimes the main body leave out the fact that the study is only in mice.

------
3JPLW
Same is true for any sort of science. We're trying to find some sort of a
story, and thus we bias ourselves towards bugs that lead to a meaningful
conclusion. Most experiments fail, but a bug can make them look like a
success.

I find it especially important if you're applying supervised machine learning
techniques. Far and away, the easiest sort of bug to make is mixing up
training and testing and holdout data (or not doing it at all). It's such an
easy bug to make because you're trying to improve the model accuracy... and a
really easy way to do that is to train and test on the same data.

------
ndiscussion
Oddly funny to me that there are data journalists out there not following the
headline advice. The average person really is doomed.

------
TomMckenny
You would have thought the meteor with Martian bacteria, the University of
Utah's cold fusion and the OPERA team's superluminal neutrinos would have
driven this lesson home by now.

------
ThomPete
Not just skepticism, but extreme skepticism. Literally assume what you are
looking at is more likely anecdotal than solid evidence or patterns and even
more likely politically biased.

------
Cypher
Skepticism requires an unlimited amount of work for verifying a chain of
evidence, if one piece is wrong then the whole body of data could be unless.
Not easy for a data journalist to standup and get a new set of data when
there's a time constraint.

Even an average person isn't even going to spend the morning verifying a
weather report demanding more data for an accurate prediction. Instead they'll
elect individuals which they trust. And it's unfortunate that they're either
wrong, corrupted or exploited.

Even today I can't even watch my old favorite youtubers without being pitched
some VPN, Web hosting or audiobook service. I feel the people I trust for
entertainment have been compromised to say and act a certain way to make some
corporate CEO happy who is manipulate numbers so some data journalist can come
to some incorrect conclusion about the world and award him some approval
rating.

~~~
munchbunny
> Skepticism requires an unlimited amount of work for verifying a chain of
> evidence, if one piece is wrong then the whole body of data could be unless.
> Not easy for a data journalist to standup and get a new set of data when
> there's a time constraint.

In theory, perhaps, but not in practice. In practice, the bar is usually at
two standards:

1\. at least two uncorrelated/independent sources of truth say the same thing
(replication)

2\. peer review turns up no clear flaws in methodology

Then your conclusions are good enough for most people. You will not be able to
completely rule out the possibility of an incomplete understanding or a theory
that is mostly right and a little wrong, but the vast majority of
untrustworthy science in the news fails at basic diligence, not at "it looks
right to the best of our knowledge but turned out wrong."

~~~
super_k_ro
> "the vast majority of untrustworthy science in the news fails at basic
> diligence"

I totally agree. I think it's due to our times, because many people just
replace religious certainties (which seem to make a come-back this days btw)
by scientific certitudes. Yet it's not how science works...

> "it looks right to the best of our knowledge but turned out wrong."

It's the underlying of scientific process and it needs to be repeated until
integrated :) With that in mind, you can hold opinions (because it's useful in
life ^^) while managing to change your own mind if a better explanation
appears (even if no one is immune to confirmation bias but we can try, can we
?)

------
stillbourne
The default position should be skepticism for everyone. Including hacker news.
I know I get constantly downvoted for skeptical view points here in the
comments, I wish it were not so but quite a few people here engage in
bullshit. Most commonly seen types of bullshit here is, ancap philosophy,
cryptocurrency advocacy, diet fads, and chemophobia.

~~~
smt88
Your impression, that HN is skeptic-hostile, is the opposite of my experience.

I browsed some of your comment history for evidence supporting your position.

I saw that your downvoted comments fit into buckets I often see downvoted
(what I observe, not what I do myself):

\- excessive profanity

\- making a factual mistake

\- saying something sarcastic

In the case of sarcasm, HN seems to dislike it either because the comment is
taken sincerely or because it "doesn't add" to the conversation.

I didn't see any evidence that you were downvoted for skepticism.

~~~
stillbourne
I use profanity, I don't see it as excessive. "Bullshit" is probably my most
often used obscenity but in the world of skepticism its used as an
interjection when someone is dead wrong in the same way a religious person
would blurt out "blasphemy" whenever someone takes the lords name in vein or
such. Also, I do get downvoted for skepticism, not for factual mistakes. Show
me where you believe I have made a factual mistake and I will show you the
academic papers that back me up. Additionally, According to "The Relationship
Between Profanity and Honesty" by Gilad Feldman et. al.
[https://journals.sagepub.com/doi/10.1177/1948550616681055](https://journals.sagepub.com/doi/10.1177/1948550616681055)
those who use profanity are typically more honest than those who don't. So why
would hacker news be against profanity if it holds the skeptical view.

~~~
smt88
I doubt many people here are "against" profanity. From only my own anecdata,
my assumption is that it makes your comment sound emotional or childish to
people.

For example, I agree with what you wrote recently[1] about diet fads, but you
did yourself a disservice by starting the comment with, "What? No! Fucking
hell."

Your position is reasonable, but it begins like a lot of angry, irrational
rants that we see on HN. I assume some people downvoted without getting past
the first sentence.

I think something similar happened when you railed against reddit[2]. I agree
with you, but you come across as cranky.

Finally, I see some examples of personal comments that are often downvoted. In
one case, you say someone is a libertarian without knowing them (and seemingly
in a dismissive way)[3]. You also have at least one post flagged for being
personal[4].

My overarching suggestion is this: HN has a very different implicit culture
than reddit. While I agree with you on almost all of your ideological
positions, I also agree with HN's downvotes most of the time.

The culture of "unemotional, value-add comments only" gives HN a much higher
signal-to-noise ratio than reddit. There are also surprisingly few flamewars
here.

If you find your emotions appearing in your comment, take a few more seconds
to re-read and edit them out. I think that in the marketplace of ideas, yours
are fairly reasonable and popular, but the tone seems to rub people here the
wrong way.

1\.
[https://news.ycombinator.com/item?id=20141441](https://news.ycombinator.com/item?id=20141441)

2\.
[https://news.ycombinator.com/item?id=19572278](https://news.ycombinator.com/item?id=19572278)

3\.
[https://news.ycombinator.com/item?id=18781643](https://news.ycombinator.com/item?id=18781643)

4\.
[https://news.ycombinator.com/item?id=18781789](https://news.ycombinator.com/item?id=18781789)

~~~
stillbourne
Conversely, do you understand how petty it sounds me? "Ew, he used a cuss.
Better down vote him." Does swearing detract from the formulation of my
argument? No. Does it add a little spice to conversation? Maybe. However, the
idea that I should not use a tone of voice that conveys shock when someone
states something outrageous is beyond ridiculous. Yes the emotion and swearing
is extraneous, but I'm not going to respond in plebeian fashion just on the
off chance I might offend someone. We're a bunch of grown ups here, no need to
pretend like this is some sort of library where we can't raise our voices a
little.

I will admit that perhaps I got too personal on occasion. Also I was not aware
of the rules for comments as I had not previously read them.

------
forgottenpass
Journalists can't even competently review and summarize a published study.

The default position for data journalism should be: don't try, you fucking
hacks.

------
yters
Should I be skeptical about the default position?

~~~
danso
Wickham doesn’t say to be “skeptical about the default position”, which
implies being skeptical of the status quo. He seems to be saying be skeptical
of what the data purports to _be_ , until you understand its origin and
collection process and etc.:

> _But until you’ve understood the process by which the data has been
> collected and gathered ... I think you should be very skeptical. Your
> default position should be skepticism._

This seems reasonable to me. Data is inherently a simplification of the
reality it purports to observe and store. Not adequately researching and
challenging the data is taking too much of a leap and faith.

------
StanislavPetrov
The "default position" should be skepticism for everyone, all the time - not
just data journalists.

~~~
SketchySeaBeast
Easy to say, hard to practice. I often don't verify that the bag of rice I
bought with a digital monetary transaction is rice until I use it for the
first time, and I don't even check it for biological agents at that time!

~~~
BackBackBack
Default skepticism doesn't mean verify everything every time. It means that
when confronted with a new claim, a claim you've not decided yet its validity,
you should abstain from taking a position until you have enough evidence to
accept or refute the claim.

In the case of your example, I gather you've bought rice before, you've
probably bought at that store before, you've used money before and you've
eaten food before. By the looks of it, you've probably cooked before with that
same kind of rice. Hence you are not evaluating the rice and all its
production line from scratch every time you buy the rice. You already
evaluated that claim and you've found that the store and their products are
acceptable to you and that is why you probably bought the rice.

In that case you abandoned long ago the default position because you already
had the required evidence to take an action. Which is the only purpose of the
default position.

The default position is not the forever position.

~~~
SketchySeaBeast
The problem is that now you've evaluated that news agency or political leaning
and you've stopped evaluating it, and 15 years later the message has changed
but you haven't noticed.

~~~
BackBackBack
Why would you only evaluate the news agency? You evaluate "from the claim" not
"from the claimants".

I know that is what people do, and that is why this article exists, to tell
them not to do that. Ironic I know. But hopefully someone will get it.

