
“Mindless Eating,” or how to send an entire life of research into question - Sindisil
https://arstechnica.com/science/2017/04/the-peer-reviewed-saga-of-mindless-eating-mindless-research-is-bad-too/
======
tryitnow
Tragically, I bet this is all too common. Wansink's mistake was blogging about
it.

This emblematic of a larger problem with how science is practiced: the
obsessive focus on p-value thresholds leads to irrational practices like
trawling data for interesting "findings."

But on a certain level Wansink was right: a data set is not completely
worthless if it showed a null result. So we need to start thinking about how
to communicate the value of data even when the null is not rejected.

One way to do this is to encourage widespread sharing of data sets regardless
of the outcome of the experiment. Maybe for a given study the data did not
show a definitive result - but does the data point to potential future paths
of study? Maybe another researcher could get ideas for new experiments.

~~~
StavrosK
> a data set is not completely worthless if it showed a null result.

Why is a null result worthless? I don't understand. It goes against my common
sense that paying more for food doesn't make you eat more. The fact that this
is not the case is valuable knowledge to me. I understand that it's not as
glamorous as "paying more makes you eat less!", but it's still valuable
knowledge that should be published.

~~~
KingMob
Null results are "worthless" only in the sense that they're harder, if not
impossible, to publish, and do less for your career.

As a former neuroscientist, I had a unicorn data set of intracranial EEG data,
and we'd spent so much time collecting it, that we were determined to find
_something_. I left grad school before we found anything of interest. I
believe my former prof eventually published _something_ on it, but I analyzed
the shit out of it, even knowing I was fishing, because so much time would
have been lost to not use it.

Fishing like this was one of the reasons I left. The pursuit of knowledge and
the pursuit of your career don't align often enough.

To me, what's unusual is not that Wansink pushed to keep analyzing the data,
but that he was called out for it. _Everyone_ seemed to be doing it when I was
in academia.

~~~
noobhacker
I feel the same way about current practice research, but find that industry
standards are even more abysmal in terms of fishing. What field are you in now
that you find an acceptable level of rigor?

~~~
KingMob
Hah, well, I returned to software dev, so I'm not too worried about research
rigor these days.

------
badosu
This is something that irritates me to the point of irrationality.

Why so many people spend so much time and resources on $name diets,
questionable research, news about how 'xyz' is good for you while 'abc' is
bad, exercises to eliminate calories in 30 days, exercises to remove localized
fat etc? All of this is shown to be evidently bullshit for decades for the
minimally rational observer.

Meanwhile the knowledge of what works is blatantly obvious: the energy
expenditure must be higher than the intake. Having a healthy diet and exercise
regimen along that is highly desirable but not strictly necessary, stating
this just because understanding the core concept is more important than
sheepishly believing the latest fad. Also that this is a process that takes a
lot of time and is not a one-off procedure but a process of learning life-
changing habits.

I'd like to remark that this is not easy at all! In fact is really hard due to
the inherent biological and societal tricks that play on our minds. But if
people are already suffering psychologically and financially with this, why
don't just try the basics?

I understand how this is incentivized by an industry that extracts money from
desperate people trying very hard to feel accepted by what society indicates
as an acceptable and desirable appearance. The irony is that the failure of
their latest hope is what makes them unable to understand that the problem is
much simpler (not easy), and only try again on a more desperate attempt with
the latest extreme measure.

At least if people would be honest enough to blame themselves for their bad
habits (lack of a healthy diet and exercise regimen) they could start the
process of accepting what they are as a result of their choices, and finally
notice that they have some agency on this.

I am not talking here about existent mental and biological disorders of course
and would not downplay their role.

Sorry for the rant, the only way I can rationalize that this exists is that
there are people that deny the existence of climate change.

~~~
CharlesW
> _Meanwhile the knowledge of what works is blatantly obvious: the energy
> expenditure must be higher than the intake._

This is like saying about the poor, "What works is blatantly obvious: Poor
people just need to increase income or reduce expenditures. Boom! Poverty
solved."

The secret is that _this isn 't the problem_. The actual problem that fat
people must solve is not _what_ to do, but _how_ to do it, both in the short-
and long-term.

> _Having a healthy diet and exercise regimen along that is highly desirable
> but not strictly necessary..._

1,000 calories of cupcake are 1,000 calories of chicken breast are, in theory,
the same amount of energy. In practice, diet and exercise considerations are
integral aspects of the actual problem that fat people must solve.

Example: For some, exercise is actually counter-productive and causes more
hunger. For others, exercise is a crucial part of success.

Example: For some, calorie restriction can be done with no particular
attention paid to the balance of carbs, proteins, and fats. For others, a low-
carb diet means lower hunger/higher satiation for longer and is a must for
success.

~~~
badosu
> The secret is that this isn't the problem. The actual problem that fat
> people must solve is not what to do, but how to do it, both in the short-
> and long-term.

You're right, it just gets to my nerves as this is a source of huge suffering
to most people and there's an industry specialized to prey on this.

~~~
CharlesW
Couldn't agree more!

~~~
badosu
I just want to point out that this is not restricted to 'fat' people, but a
great majority of people who feel a huge pressure to adhere to a physical
appearance standard without the proper education of how to get there and
maintain it.

------
cm2012
PSA I always post into weight threads: Every legitimate long term study of non
surgical weight loss shows that it doesn't happen for the vast, vast majority
of people.

1) ["In controlled settings, participants who remain in weight loss programs
usually lose approximately 10% of their weight. However, one third to two
thirds of the weight is regained within 1 year, and almost all is regained
within 5 years.
"]([http://www.ncbi.nlm.nih.gov/pubmed/1580453](http://www.ncbi.nlm.nih.gov/pubmed/1580453))

2) Giant meta study of long term weight loss: ["Five years after completing
structured weight-loss programs, the average individual maintained a weight
loss of >3% of initial body
weight."]([http://ajcn.nutrition.org/content/74/5/579.full](http://ajcn.nutrition.org/content/74/5/579.full))

3) Less Scientific: [Weight Watcher's Failure - "about two out of a thousand
Weight Watchers participants who reached goal weight stayed there for more
than five years."]([https://fatfu.wordpress.com/2008/01/24/weight-
watchers/](https://fatfu.wordpress.com/2008/01/24/weight-watchers/))

4) [The reason why it's impossible seems to be that although calories in <
calories out works, the body of a fat person makes it extremely difficult
psychologically to eat
less.]([http://www.nytimes.com/2012/01/01/magazine/tara-parker-
pope-...](http://www.nytimes.com/2012/01/01/magazine/tara-parker-pope-fat-
trap.html?_r=2&pagewanted=all)) This is borne out by the above data.

5) [The only thing that does seem to work in the long term is gastric
surgery.]([http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1421028/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1421028/))

Moreover, you won't find any reputable study on the web where the average
person lost 10%+ of their body weight and kept it off for five years. Not even
one.

~~~
drewcrawford
> Moreover, you won't find any reputable study on the web where the average
> person lost 10%+ of their body weight and kept it off for five years. Not
> even one.

Of course there is:
[http://www.nwcr.ws/Research/published%20research.htm](http://www.nwcr.ws/Research/published%20research.htm).

Here's a good question: other than the fact that these people lost weight,
what is identifiably unusual about them?

~~~
npsimons
Was going to link to the National Weight Control Registry, thanks! I'll just
add that all those studies in GP seem to prove is that A) weight loss
_programs_ (especially fad diets) don't work and B) it is a psychological
issue. There are plenty of people on MFP, /r/loseit or just counting calories
themselves that have successfully kept off weight for years. I'm one of them.

In case someone out there is serious about losing weight and not making
excuses, here's how you calculate your actual TDEE:
[https://www.reddit.com/r/leangains/comments/2rv09z/this_is_h...](https://www.reddit.com/r/leangains/comments/2rv09z/this_is_how_you_calculate_your_tdee/)

~~~
js2
Many times I'll read a story of someone who lost weight and kept it off. And
then they detail their pre weight-loss diet and I think, well of course you
were overweight. You were inactive and had a terrible diet (sugary drinks,
processed foods, etc). You started getting some exercise and learned a few
things about nutrition and the weight fell off.

But then there are others who seem to do everything right and are over weight
in spite of that.

For example. I was never overweight as a kid and relatively active. In college
and for the start of my career, I stopped being active and my diet was awful
(e.g. I thought a large Jamba Juice smoothie and a carrot cake was a healthy
breakfast choice). My weight ballooned up to almost 190 lbs (20+ lbs
overweight), my blood pressure went up, I started having rosacea.

I started running and fixed my diet. Quickly my weight dropped down to 150 and
I've kept it in the 140-150 range for over a decade. The other health issues
cleared up as well. But it wasn't hard work for me. Being thin is my natural
state if you will, and I had to do everything wrong to stay overweight.

My wife meanwhile continues to struggle with her weight. She's successfully
lost weight through extremely diligent calorie counting, but after a year or
so she starts to put it back on. I have never counted calories. Our diets are
similar (in kind, not quantity of course, she eats much less than me). She is
active, but not quite as active as me. So similar diet and life styles, but my
weight stays off and hers does not.

Hereditarily, no one in my family is over weight. There is obesity on both
sides of her heredity.

And I see this playing out in our kids. My son has an athletic build and will
probably never have weight issues. My daughter takes after her mom and it will
take a life time of diligence for her to remain at a healthy weight.

It seems that some people are optimized for famine, and some for feast. :-(

Obviously there are a lot of factors involved in the growing obesity crises.
But I feel for people who struggle with their weight despite doing all the
right things, I really do.

~~~
atomical
The food tastes too damn good! I've only been overweight because of binging
and poor eating. I've never eaten in a normal, healthy way, and gained weight.

Calories are such that if you screw up once per week (birthday party, company
event, family dinner) that could mean you gain weight if you eat regularly the
rest of the days.

------
panglott
'Why did peer review not catch this? “Because peer review doesn’t do this,”
Heathers told Ars. The point of peer review has always been for fellow
scientists to judge whether a paper is of reasonable quality; reviewers aren't
expected to perform an independent analysis of the data. ...In fact, without
open data—something that’s historically been hit-or-miss—it would be
impossible for peer reviewers to validate any numbers.'

It doesn't seem unreasonable to think that the peer review process should
include a statistician or someone who can review the statistics.

~~~
andrewla
> In fact, without open data ... it would be impossible for peer reviewers to
> validate any numbers

Even this is not sufficient -- the fact is that this is a methodological
statistical error, not a mathematical statistical error.

The mistake is thinking that a dataset alone can yield data supporting a
hypothesis derived from that dataset, rather than deriving a testable
hypothesis from an existing dataset, and then gathering new data to validate.

I would say the only way that you could publish results derived from an
existing data source is if you also published all the null results you got
along the way when examining the data set; but this is not really a feasible
thing to print in a publication; either the list would be very long and thus
it would be clear that some sort of p-hacking was involved, or the list would
be short and accidental or intentional omission would be suspected.

This dramatically reduces the amount of useful results that can be squeezed
from a dataset, which is unfortunate, as many of them are hard to gather in
the first place. It might be necessary to protect these datasets better -- to
restrict access conditional on specifying the hypothesis being tested with the
requirement that all results, including null results, be recorded, even if
only in summary form if the results are not interesting enough to be accepted
for publication or to be worth the effort of composing into a quality research
paper.

~~~
danso
> _Even this is not sufficient -- the fact is that this is a methodological
> statistical error, not a mathematical statistical error._

That said, Wansink's papers had plenty of mathematical statistical errors that
were evident without access to the original data, as documented in the
mentioned "Statistical Heartburn" preprint:
[https://peerj.com/preprints/2748.pdf](https://peerj.com/preprints/2748.pdf)

~~~
andrewla
Very much so -- I had only read the first part of the article when I wrote
this reply, having read the whole thing now (and the linked pizza paper) I'm
more than a little horrified at what can, in the most optimistic case, be
described as extreme carelessness.

I have a pre-existing bias against papers relating to nutrition science, and
this looks worse even than I expect. I would love to see a treatment that
looks at especially noteworthy research in the area that passes a rigorous
methodological review, much less has been replicated with any reliability.

------
tonmoy
If the academic world let scientists publish their validation of the null
hypothesis, then they wouldn't go out of their way to "deep dive" into data to
look for other hypothesis to prove!

------
jm__87
Feel like we really approach the whole dieting process incorrectly. People eat
to fulfill a need. They are either hungry and eat to no longer feel hungry or
they eat because it produces positive feelings (whether this is to negate
negative feelings or just to enhance your already good mood).

Hunger is your body signalling your brain that you need nutrients. If you eat
foods that cover all your nutrient requirements for a low number of calories
and your body is taking up these nutrients properly, then you should have no
problems. Thus it makes sense to inform people which foods they need to be
eating as well as educate people on which disorders can cause poor nutrient
uptake.

If you suffer from stress and you eat high calorie foods to feel better,
changing your diet is likely going to exacerbate your stress, not make you
feel better. I feel this is the case for many who are overweight.. stress
management strategies should be the first priority as you won't be able to
handle changing your diet for the long term until you get your stress under
control.

~~~
temp246810
This doesn't even begin to get into the effects different macronutrients have
on you. It's a complicated subject that has reached religion-like levels of
zealotry.

------
js2
The article makes Wansink seem negligent and/or incompetent. Also, the
embedded "CBS This Morning" video is cringeworthy. It appears that the
recommendations in his book amount to "study normal weight people; do what
they do." Isn't that fraught with survivorship bias? Do the recommendations in
his book control for people that have the same habits but still end up
overweight?

Anecdote: my household is a family of four. Myself and my wife, and our two
teenage kids, daughter and son. Since we live together, the answers to the 10
questions on [http://www.slimbydesign.com/get-
scored/](http://www.slimbydesign.com/get-scored/) are the same for the
household. My son and I are normal weight. My wife and daughter are
overweight. Conclusion: weight has very little to do with the design of your
kitchen. Caveat: this conclusion has not been peer reviewed and is not
scientifically sound.

~~~
npsimons
> Isn't that fraught with survivorship bias?

It is, but it's not a completely invalid starting point. Even better is to
study people who were once fat, but lost weight and keep it off. This is what
the National Weight Control Registry does. It's similar to studies that have
tried to pin down why marriages fail. Most studies focus on what went _wrong_
in failed marriages, but some of the most helpful suggestions have come from
what people who stay married do.

> My son and I are normal weight. My wife and daughter are overweight.

Do your wife and daughter eat the same portions as you and your son? It's a
common complaint of women, who are on average shorter than men and therefore
need fewer calories, that they can't eat as much as their significant others.

~~~
js2
> Do your wife and daughter eat the same portions as you and your son

No. I expand on that a bit here -
[https://news.ycombinator.com/item?id=14196631](https://news.ycombinator.com/item?id=14196631)

------
draw_down
The Last Psychiatrist talked about this sometimes. The problem is not just
that the original studies need to be rescinded, or discredited going forward.
The problem is what do you do about everything that cited them, that took the
knowledge in the faulty papers and used it as a foundation for more knowledge?
The faulty ideas in the paper have permeated out to researchers and
(sometimes) the broader culture generally, what do you do about that?

[http://thelastpsychiatrist.com/2009/10/the_problem_with_scie...](http://thelastpsychiatrist.com/2009/10/the_problem_with_science_is_sc.html)

------
astrobase_go
I think what's needed is a strategy for making actual data analysis an easier
and more visible component of the peer-review and publishing process.
Submissions are typically sent to three people for review, and the decision to
accept or reject is made based on the feedback or criticism given. If the
reviewers are busy, overworked professors, they may not have the time to
really perform a deep dive into the data and conduct an independent analysis.
Furthermore, it's likely these submissions don't include enough data for a
proper statistical review.

This is the 21st century, where most if not all reputable journals have an
online presence and submission portal. It would be great if authors had to
upload their anonymized data sets in a common format (.csv?) that can easily
be imported into statistical analysis packages (ex: Minitab, read into R,
etc). Journals provide analysis and test recommendations, reviewers run the
tests, upload their independent analysis results as part of the review
process. The idea here is to produce something like an auditable paper trail.

There has to be some sort of solution for this problem, especially in 2017.
Hiding behind shitty (or worse, deceitful) data analysis shouldn't be
possible.

------
Animats
_" Trawling through data, running lots of statistical tests, and looking only
for significant results is bound to turn up some false positives."_

That's most of the field of macroeconomics. It's all data analysis, not
controlled experimentation.

~~~
mcguire
I was just about to ask, isn't that how data science works?

~~~
beagle3
Proper data science (for example, bayesian analysis done right) takes that
into account, and would generally not score random finds as remarkable.

But doing that is hard, so most people don't even try ...

yes, practically, that's how modern data science works.

------
kwhitefoot
> Many scientists receive only cursory training in statistics,

This seems to be true (at least I have many younger colleagues who are
woefully ill informed about even basic stats.) but it's very strange.

Most of what I learned of statistics, including tests for significance and
regression, I learned in senior high school at the age of 17.

Learning not to search of correlations like this is pretty much equivalent to
ensuring that you do not over fit your data set which is surely such
conventional wisdom that no one getting a scientific education in the last
sixty or seventy years should have missed it.

I'm aware that there is a lot of highly sophisticated statistical analysis
that I don't know about but things like p value are not among them.

------
candiodari
Inside universities, it does not seem realistic that this will lower people's
opinions on the value of social science statistical research, or in the
related fields of psychology (esp. management/organisational psychology), or
even all of the humanities.

That opinion is already "reject it all".

Even when it comes to medicine that opinion is strongly represented. A few
people even ascribe the placebo effect to massive and widespread statistical
error.

