
The Problems with Food and Exercise Studies - eagerToLearn
http://www.nytimes.com/2016/08/11/upshot/were-so-confused-the-problems-with-food-and-exercise-studies.html
======
yk
I am no expert on non-physics sciences, but recently I am starting to worry
that the way we do science today depends on ideal experimental and theoretical
conditions.[1] In physics we see 2 sigma events vanish all the time, just last
week we have seen the disappearance of the di-photon excess at the LHC. That
was an excess that two experiments did see independently to a significance
better than p < 5%, with well controlled statistics and well understood
experimental set up. Similar the Auger anisotropy [2] was on the cover of
Science and did just vanish. Again an effect with p < 5%, using blinded data
analysis. My understanding is, that this data analysis is quite a bit better
than the standard in other fields, and it does not really work reliably in
physics, a field that is to some extend defined by ideal experimental
conditions. (The saving grace for physics is, that the Standard Model and
General Relativity tie different experiments together. So an experiment in
conflict with expectations is immediately in conflict with a lot of other
experiments.)

To be clear what I am saying, I do not think that other sciences are a waste
of time in general. But I think that their methods are too much inspired by
physics. I think that we need a much more cautious approach to the scientific
method and a lot more robust tools. In a way, it looks to me like building a
car by taking inspiration from Formula 1, when the task at hand is crossing
rivers and navigating rough terrain.

[1] Previously I assumed that it is just complicated, that all I see from
studies is the headline "Coffee is/is not healthy" and if one would look into
the studies I would find that a better picture is Coffee has some positive and
some negative effects.

[2] An anisotropy in the arrival directions of ultra-high-energy cosmic rays.
[https://arxiv.org/abs/0711.2256](https://arxiv.org/abs/0711.2256)

~~~
jsprogrammer
2-sigma events should vanish all the time. p < 5% just means that you see it
(vanishment) happening less than 1-in-20 times. If you have hundreds of such
events, you'll have dozens of vanishings.

~~~
mamon
I'm no expert on this but isn't p < 5% riddiculously low threshold? I would
rather set standard at p < 0.5% or even 0.05% This would probably solve
"replicability crisis" in "soft" sciences like psychology.

~~~
stdbrouw
The problem is that then you will never find anything. There's always a trade-
off between finding stuff that's not there and not finding stuff that really
is out there.

------
rdmcfee
There are two core issues with nutrition science.

The first issue is that it's extremely difficult to run a long term controlled
study on nutrition. Adherence to diet interventions is difficult to measure
and it drops off over time. As a result most nutrition studies are
epidemiological cohort studies which are not particularly good at determining
causality.

The second issue is a question of funding. Diet and exercise interventions are
not something that are typically monetized be the pharmaceutical industry. As
such there is little private sector funding going into nutrition studies.
Couple that with the fact that they're extremely expensive and you have a
serious deficit in studies.

~~~
andrei_says_
Once I found out that food calorie count today is still determined by
literally burning the food in a calorimeter and measuring the heat, a practice
started in the 1800-s, I realized that I am on my own.

I get that the laws requiring calorie content on labels have lead to finding
an efficient way to measure and provide the numbers, but it is obvious to me
that the measuring system is completely broken or very inaccurate at best.

[http://www.scientificamerican.com/article/how-do-food-
manufa...](http://www.scientificamerican.com/article/how-do-food-
manufacturers/)

~~~
squidfood
I'm not sure what your problem with this method is, given that the amount of
heat given by burning is the _definition_ of a food calorie.

That's like saying "once I learned that voltage was measured by the deflection
of a coil in a current (a practice started in the 1800s)" or "once I learned
that heat was measured by the expansion of a liquid in a glass tube..."

You could argue that the heat definition of calorie doesn't really reflect the
health function of the food (for example, maybe glycemic index is more
useful), but I'm not sure why complaining that a calorie isn't a calorie is
really insightful.

~~~
x0x0
Your body isn't actually burning the food. Therefore, why is this an important
measure, other than it is easy to measure?

Bioavailability / metabolizable energy varies not only in the small, ie
composition and cooking of food, but in the large, as long run diet choice
changes gene expression.

~~~
squidfood
Actually, respiration _is_ burning the food, it's literally the same energy
release, and the methods of calculation take into account the portion of the
food that's burnable but non-digestable. If you're saying that the corrections
for indigestible food are the subject of scrutiny, that's true, but that has
nothing to do with the fact that "burning" is a problem for calculating
caloric content.

I used to work in aquaculture, where we measured the amount of food going into
a fish (via the bomb calorimetry that we're talking about, directly), the
amount of energy spent by the fish in respiration (by measuring carefully
increases in water temperature in their tank), and then we measured how much
energy the fish had at the end (by burning the fish).

And quite simply, calories in was equal to calories out. It was a good
relationship within very acceptable experimental error.

Now of course, the fish were growing very rapidly and we could control their
food exactly. Humans, growing over a longer period, can be subject to
different metabolic pathways, and we can't measure their respiration directly
of these pathways [ _Edit: this is where the gene expression you mention comes
in_ ] - this is a far, far bigger source of error than the calorimetry, so
calories alone can be less predictive of outcome. But this doesn't take away
from the basic definitions of thermodynamics.

~~~
PeterisP
1) respiration is burning the food, but it's not burning _all_ the food;
"calories in" refers to calories absorbed, not calories put in your mouth; and
the proportion between those two varies depending on all kinds of things,
including e.g. gut microbiome and temporary changes to that due to various
drugs; so if you strictly count calories put in mouth you still get a hard-to-
measure variability in actual "calories in" (unless you put a respirator on
the subject and measure it that way).

2) All other things being equal, changing "calories in" will make you
lose/gain weight. The trouble is, all other things are not equal - simply
changing how much you eat will _significantly change_ those other things, it
has an effect on your metabolism and eagerness for physical activity, thus
directly affecting also "calories out" if you don't carefully monitor that and
work to keep that stable.

3) All other things being equal, changing "calories out" will make you
lose/gain weight. The trouble is, all other things are not equal - simply
starting/stopping working out _will_ significantly change your natural
appetite, how hungry you are and how often you're hungry, and which types of
food you have cravings for, thus directly affecting "calories in" unless you
carefully track what and how much you eat everything and actually do keep that
schedule exactly the same.

~~~
squidfood
Yes, this is a good partial list of all the "bigger sources of error" that I
mention above!

------
tigerBL00D
Let's face it, the majority of the population lazily look for simple formulas
along the lines of "(Don't) do this, and you'll get that". Media panders to
that when drawing conclusions from studies often leaving out most or all of
the important details. 99% of folks out there hear of these studies via media
and not by reading research papers. Meanwhile most simple formulas of this
kind are not worth the energy it takes to draw the pixels given the profound
variety of genetic and environmental factors at play.

I find the suggestion at the end of this article truly pathetic: "Some medical
experts say the problems with lifestyle studies are so overwhelming — and the
chance of finding anything reproducible and meaningful so small — that it
might be best to just give up on those questions altogether." This is typical
big media bullshit pandering to the inert.

I mean come on... How about we don't give up on questions? How about we learn
to ask the right questions instead? How about we spend more time training and
studying our own bodies than we spend reading our Facebook feeds?

~~~
stdbrouw
I don't think you could've written anything more arrogant if you tried.

> the majority of the population lazily look for simple formulas

Great! After all, it's not that different from "a good programmer is a lazy
programmer" and "keep it simple, stupid" \-- if at all possible, look for
simple solutions and quick fixes.

> This is typical big media bullshit pandering to the inert.

"Epidemiology--is it time to call it a day?"
([http://ije.oxfordjournals.org/content/30/1/1.full](http://ije.oxfordjournals.org/content/30/1/1.full)),
an essay that comes to pretty much the same conclusion as this article, was
published in 2001 in the International Journal of Epidemiology. As far as I
know the International Journal of Epidemiology is not big media.

> I mean come on... How about we don't give up on questions?

Rhetorical questions don't count as arguments.

> How about we learn to ask the right questions instead?

Observational research is intrinsically very, very hard. Statisticians are
working at making it more reliable (especially in the context of causal
inference: [https://www.hsph.harvard.edu/miguel-hernan/causal-
inference-...](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-
book/)), but there's only so much you can do with data that is full of noise
and confounders.

> How about we spend more time training and studying our own bodies than we
> spend reading our Facebook feeds?

You don't get to decide other people's priorities in life.

~~~
hinkley
> if at all possible, look for simple solutions and quick fixes.

Oh boy. I'm not OP but you and I could easily have the same altercation. Quick
fixes and simple solutions are two different tools, and conflating the [two]
leads to endless disagreements and hurt feelings.

Pascal and de Saint-Exupery both have famous quotes that apply to the
distinctions between the two.

~~~
CodeMage
I know I can look those quotes up, but that endeavor will most likely end up
ramifying into a lot more procrastination than I can afford right now. Could
you post those quotes?

~~~
hinkley
A letter from Blaise Pascal to a colleague starts, "I apologize for the length
of this letter. It would have been shorter if I'd had more time."

Exupery has the one about perfection being achieved when there is nothing left
to take away. I confess you have to unpack that one a bit to turn it into a
commentary on simplicity and how it's challenging to achieve.

There's the one attributed to Einstein about 'as simple as possible but no
simpler', and for some reason I occasionally swap the two in my head. Any
problem is easy to solve if you misunderstand it properly.

------
jacques_chester
Based on seeing a lot of exercise literature, back when I took an interest:

* Small sample sizes

* Self-reporting and diary studies

* Many uncontrolled variables

* Inconsistent interventions (not isocaloric, not volume-equivalent or tonnage-equivalent etc)

* Impossibility of performing blind trials to control for placebo

~~~
astazangasta
The whole notion of 'controlled variables' is a ridiculous exercise. At some
point we have to cop to the fact that biological interaction is too complex
for our shitty parameterization. We're carving mountains with a tack hammer.
Maybe there is a better math out there that can manage, but I haven't seen it
yet.

~~~
jacques_chester
Two things.

First, "it's hard and we can't control everything" isn't the same as "it's
impossible to control anything".

Second, large sample sizes are preferred exactly to help even out the
variables that are uncontrolled or, especially, unknown.

~~~
astazangasta
I'm not saying "it's impossible to control anything", I am complaining about
the idea that the world can be represented as a set of variables, that
everything can be quantified and broken down into axes, and this
parameterization is a useful or helpful way to model the vast interacting
complexity of a biological system. If you have ever tried to do PCA on a gene
expression dataset, you know my pain. The exercise is futile; you are
attempting to produce a description of the world based on a presumption that
is wholly inadequate.

~~~
jacques_chester
My point was that you're introducing the nirvana fallacy: rejecting all
attempts because they cannot satisfy a perfect outcome.

That it's impossible to perfectly model reality is not news in any scientific
field.

In any case, I was talking about grossly observable variables such as "does
this person exercise at all?", or "do they smoke?" or "do they drink?". You'd
be surprised how many studies don't bother to check simple factors like these.

------
pessimizer
This is just a problem with capitalism. Food and exercise studies are turned
into health products, and a minimum amount of scrutiny results in the widest
range of products. Advertising-supported media can only write about those
products positively or not at all, because it takes money from them, or may
potentially in the future. Rewriting a press release is cheap.

There's also the mentality that every aspect of what you consume must be
scrutinized intensely, or else you risk destruction. We probably get that from
our more adventurous forebears eating poisonous stuff, through religion.

------
beat
Michael Pollan's writing on "magic ingredients" is far more telling than these
studies. People want a magical solution that will allow them to be "healthy"
without changing their lifestyle.

------
mr_tristan
At least I'm starting to read about inconsistent measurements. I'm always
baffled how the only really consistent metric in use is body weight or BMI,
which is pretty useless to pay attention to. I mean, you can just starve
yourself and your BMI will go down whenever you want to take a test.

Where I get pissed off is that health insurance companies will create policy
around research based on shitty metrics. They really only seem to be capable
of understanding what might help when things go REALLY wrong - as in you
require constant drug use or surgery, because hey, they have patient data on
that. But in terms of preventive medicine, insurance companies can't figure
out what to do. Thus, it's completely random as to what you may or may not get
covered for preventative techniques - like doing physical therapy to _prevent_
surgery instead of recover from it. And if you have to pay out of pocket for
this sort of stuff, damn... it gets expensive these days.

------
scythe
As far as I'm aware, there are less than five studies -- _total_ \-- which
have been carried out on a reasonably representative sample of the population
(i.e. not military service-members), which studied diet patterns, and which
lasted strictly longer than two years. Most people live much longer than two
years, so this would seem to be a systemic weakness.

But people lived before peer-reviewed research. They collected information
somehow and used it somehow and it worked to varying degrees of success, which
has been reflected in the rise and fall of various societies and in the
quality of life in those societies. The challenge is not to abandon peer-
reviewed research, but to integrate it effectively with other sources of
knowledge.

------
known
If you don't read a newspaper you are uninformed. If you do read a newspaper,
you are misinformed.

------
astazangasta
An unmentioned problem is the idea of norms, a moddishness that is sometimes
equated with 'science', whatever that is. Humans - species, biology - is not
normative, it is diverse. Genetic diversity means two individuals are not just
variation from a mean, they are different. The attempt to produce summations
of what a human should be is flawed at the root. But we are too attached to
Gaussians and standard deviations, so I imagine we'll continue beating our
heads against this wall in a vain attempt to create a knowledge that can't
exist.

~~~
chewyshine
Doesn't the diversity get represented in the standard deviation of the
distribution? The Gaussian distribution is more than just a convenient
statistical assumption. Normal and power law distributions seem to be
consistent with a remarkable array of human diversity and central tendencies.

~~~
astazangasta
Yes, the central limit theorem is great because it conveniently masks
complexity and lets us represent diversity as outliers on a spectrum. But the
basic model of a Gaussian - of a central tendency and deviations from it -
doesn't represent the biological reality, that the "central tendency" is only
the product of our summary, and that there is no gross general 'generating'
principle underlying it.

That is - the outliers often differ from the mean in a biological system
because they are fundamentally different; they have different underlying
biology, and while we might be able to place everybody on the same normal
distribution, any conclusions we draw based on that normal won't apply to
individuals whose private variation makes them behave differently.

------
kbenson
The solution is coming. Life-streaming. Don't rely on self reporting, just
track everything and pick out the data that's relevant. Software to identify
specific markers in the video streams, such as the presence of food, eating,
drinking, exercise, etc will make this much more feasible.

A multi-year study with thousands of participants is possible. If you aren't
worried about perfect data, wait a decade and start consuming what I'm sure
will be many public life-streaming feeds.

~~~
hyperpape
That seems...somewhat subject to selection bias.

~~~
zardo
All population studies are, it needs to be accounted for and minimized, but it
doesn't have to be eliminated.

~~~
hyperpape
With regard to the status quo, you could say: "All population studies are
subject to incomplete or inaccurate data. It needs to be accounted for and
minimized, but it doesn't have to be eliminated".

I guess livestreamers are another data point, so in a perfectly pure
framework, they can't actually make your epistemic situation worse, no matter
how bad the selection bias is, but I don't see how they're going to make the
situation that much better.

~~~
zardo
I think they do help. For constant dollars, each study has to balance the
detail of the data captured against the size of the population included.
Anything that allows more detailed data to be gathered from a larger
population is helpful.

While that population is going to skew wealthy if the study involves the
participants making a purchase, that can be counteracted by seeking out under-
represented populations and asking and/or incentivising them to participate.

~~~
bluGill
>I think they do help. For constant dollars, each study has to balance the
detail of the data captured against the size of the population included.
Anything that allows more detailed data to be gathered from a larger
population is helpful.

Not exactly because we don't know all the factors to control for. You end up
with a bunch of correlations but often it isn't clear why. Are people who eat
X foods Y because they eat that, or because the common genetic background of
people who eat X are Y? Or maybe it is because before they were born their
parents did Z?

------
a3n
You should probably eat things that are good for you, like fruits, vegetables
and protein, and not eat too much that is bad for you, like boxes full of
ingredients. And don't worry about coffee and a little bit of alcohol.

~~~
mhurron
> boxes full of ingredients

What the hell does this mean?

~~~
hanniabu
I think he means processed foods, which usually come in boxes and have a never
ending list of ingredients which are 50% sugar/syrup/salt and 25%
chemicals/unknowns.

~~~
dagurp
God forbid we eat chemicals

~~~
a3n
I love chemicals. I try to eat a variety of chemicals every day. But I want to
eat the ones that are good for _me_ , not the ones that are designed to keep
things in a box from rotting.

------
YeGoblynQueenne
That is an interesting aritcle but it fails to address the most important
question: can we safely eat bacon?

------
chatmasta
I really don't understand why people have so much trouble staying fit. Granted
if you're already fat, you have a lot of work to do to get back into shape (I
say "back into shape," because nobody is born fat). But the formula for
_maintaining_ a healthy body is really pretty simple.

\- Avoid junk food (if it comes in a vacuum sealed bag, it's junk food)

\- Avoid sodas, juices, and other high sugar drinks. If you choose to indulge
yourself, don't finish the whole thing - the portion is probably too big
anyway.

\- Exercise at least a few times a week. Meaning, break a sweat and keep it up
for at least 30 minutes, preferably an hour. Eat (or drink) protein
afterwards.

\- Only eat when you're hungry. Stop eating when you're full.

\- Listen to your body.

~~~
Retric
When your job 'forces' 10+ hours of inactivity a day in the office + commute
and the only house you can afford requires driving everywhere, and cheap
entertainment is passive TV/Computers. And most food is minimally nutritious
and pumped full of empty calories so your body says eat more I need X.

It's easy to see how people get out of shape.

~~~
maxxxxx
There are so many bad lifestyle choices our society accepts as normal so
people get overwhelmed by all the changes they should make

\- Long work hours, mostly sitting

\- Long commutes

\- Portion sizes way too large (in restaurants I am often full after
appetizers)

\- Eating at the desk, eating late at night, skipping meals

\- TV watching

\- Nothing reachable by walking

\- Not enough sleep

\- Not enough vacation

\- Most foods in stores that are labeled "healthy" are not healthy

You need a certain level of stubbornness to realize that the generally
accepted lifestyle is so far away from reasonable that almost everything
that's considered "normal" needs to be changed.

~~~
Retric
Exactly, I notice people that become unemployed tend to lose weight, sleep
better, get in shape ect not because they try but because the 9-5+ grind
promotes a lot of bad habits. As a friend put it, I get 2 hours of 'me' time
every week day, I don't spend 1/2 of that working out.

That said while the default is bad you really can live a healthy lifestyle
without to much work. It just takes planning to avoid spending 10-15+ hours a
week in a car etc.

------
Aelinsaar
It's a shock when you try to switch mental gears from reading something like a
HEP paper, talking about 3 sigma like a big "Well, it could be interesting,
probably not." Then you read a front-page "study" about something involving
nutrition and "Five people tested, effect confirmed!! Everyone eat Goji
Berries!!"

It's mindbogglingly bad, non-science.

~~~
psycr
Depends on the effect size.

You probably only need n = 5 for a confident answer to the question "What
happens when you are shot in the head?"

~~~
fullshark
Studies of the form article mentioned helped solidify the link between smoking
and cancer also.

People are looking for smaller effects than before, and the incentive to
publish any and all significant results is just so great that you get tons of
noise.

------
webwanderings
Did NYT just confused the whole lot of the same confused people, by not
providing anything of value in this article? Couldn't they have just published
a stand-alone headline, which would have served the same purpose?

