
Poor Man's Scientific Method - rumcajz
http://250bpm.com/blog:35
======
ignostic
I was happy to find this, because I've had a number of similar debates - some
with very intelligent people! My appeals to the mechanism of action went over
everyone's head and convinced no one. Citing studies got us lost in the weeds
as they asked questions about the study that I didn't know the answer to. (Who
paid for it? Did they adjust for ___? Did they try ___?)

I agree with the problem, but I don't agree with several points. I'm also not
sure about the solution.

> "Today, it's common that a licensed doctor prescribes a homeopathic
> medicine, which is scientifically known to be ineffective."

Are there really? How common is "common"? This is the first I'm hearing of it,
and I'd like more than the author's word. Perhaps a study? :)

>"the profession is regulated by the state and the doctor obtains his license
only after finishing a school (where he is taught scientific method along with
any specific knowledge needed in his field of expertise), having some actual
hands-on experience and taking the Hippocratic oath."

This is actually one of the problems - our culture's over-reliance and
expansive trust in state-mandated licensing. The state is simply not qualified
to say who can do what or whether they're actually competent in a field,
especially once we start talking about specialties.

Because the state lacks the time and experienced professionals in a given
field, we came up with certification boards. There are some great boards made
up of real-life specialty experts who certify doctors as being competent in
their little niche.

Unfortunately, this gets us into the same problem: how do I know which boards
I can trust? I guarantee there are homeopathy boards that certify doctors.
Looking into the legitimacy of a board is just as hard as looking into the
legitimacy of a paper.

The answer has to be education - anything you can do, the "alternative
science" crowd and profiteers can do, too. Unfortunately, neither I nor the
essay's author seem to have a great cure-all answer for people who don't have
the time, interest, or intellect to educate themselves.

~~~
VLM
"How common is "common"? This is the first I'm hearing of it, and I'd like
more than the author's word."

They're scientifically proven to be medically ineffective although for people
with psychological non medical issues they are scientifically proven to reduce
anxiety. Daughter's got a cough. Moms freaking out. Doc verifies its not
bacterial so there's nothing to do but chill out, these viruses take care of
themselves, rest and hydrate. Moms freaking out. You treat MOM not the
daughter with a harmless magic potion. Mom is now calm, that makes daughter
calm and unstressed, and thats medically proven to help or at least not harm
someone who needs hydration and rest, and issuing poorly distilled water is
medically proven to be much safer for everyone involved than giving
antibiotics for a viral infection.

I'm a pretty strong supporter of homeopathic treatment of viral illnesses, at
least in comparison to the US (edited: informal) standard of antibiotic
treatment. Its enormously safer than antibiotics and its just as effective WRT
virus infections. Just make sure its really viral in nature not just a poorly
tested culture test in the lab.

(edited to explain I'm not talking theory here; my daughter's pediatrician has
recently switched from antibiotic treatment of minor viral infections like
coughs and colds to homeopathic treatment... when I found out from my wife
what my daughter was prescribed, I was momentarily furious, come on lady I'm a
hard science guy don't BS me with homeopathic stuff like that, but I thought
about it for about five minutes and decided to high-five her next time I see
her, it really is genius in that people too dumb to understand antibiotics
have no effect on viruses are too dumb to know homeopathic stuff doesn't work,
so the net result is positive)

I've been trying to figure out how to convince idiot industrial farmers that
homeopathic water might be a better idea than pumping the livestock full of
antibiotics. The problem is in filthy industrial farming operations, the
livestock all die without antibiotics because ... drumroll ... filthy
industrial farms are filthy.

~~~
saosebastiao
Thanks for the insight. I've never thought of the possibility of treating the
stressed-out parent with homeopathy. It seems brilliant!

My only problem is how does the doctor know that will calm the nerves? If my
doctor prescribed my kid homeopathic medicine, I would never go to him again.
Placebo only works when the people don't know they are being duped.

~~~
dllthomas
_" Placebo only works when the people don't know they are being duped."_

It's my understanding that this is (surprisingly!) not the case.

------
andyjohnson0
_" An obvious example: Sokal hoax. Physicist Alan Sokal have written an
scientific article permeated with most obvious scientific nonsense and
deliberate vagueness and managed to have it published in a scientific
journal."_

The journal was _Social Text_ , a "cultural studies" journal. At the time,
according to the same wikipedia page referenced in the article, _Social Text_
did not practice peer review.

The fact that Sokal was able to get his paper published tells us nothing about
the value of peer review in science

~~~
alaaibrahim
well, you can look up Schön scandal[1]. It would be a better example.

[1]
[https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal](https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal)

~~~
rumcajz
I wasn't aware of that. Thanks for the link!

------
aristus
Simple rule, less about trust and more about science: the paper must be at
least 5 (five) years old and have one (1) independent confirmation before I'll
consider reading it.

It's possible to read the original papers and look for obvious signs of
trouble, eg small sample sizes, improper randomization, etc. Basic science &
statistics is not rocket science.

But the independent conformation rule is a really good filter. If, after 5
years, there's no follow up, there's good odds the paper was wrong. Or even if
it's right, whatever it claims isn't considered important enough to follow up
on. Even in the case of outright fraud like the Lancet autism article, the
lack of independent studies reproducing the claimed correlation would have
been a red flag.

~~~
fractallyte
Unfortunately, some very important papers wouldn't make it past this test...

The startup I'm working with has developed a unique, revolutionary product.
(Unusually, this is a true statement - not marketing.) There are a number of
published papers, and 8 year follow-ups. But journals won't even publish the 5
year follow-ups!

Why? Because it is a _genuine_ paradigm-changer. We can't get independent
confirmation, because no one wants to touch it. It threatens a global multi-
billion dollar industry.

There's a 'mafia' in almost every profitable industry. Until I joined this
startup, I had no notion that this niche was so full of medical practitioners
readily sidestepping the Hippocratic Oath for the goal of profit.

~~~
aristus
That's fine, as far as John Q Layman's life is concerned, cutting through
false positives with minimal effort. There will be false negatives with any
rough filter. It sucks for you, but that's your fight to fight.

------
JoeAltmaier
About the placebo effect: its not about what you know, its about what you
feel. Enter a bakery, smell the cinnamon and yeast: your blood sugar responds.
Doesn't matter if you know this is a placebo effect, it still happens.

So it's consistent that people in a non-blind test can still benefit from the
placebo effect. Just going through the motions can affect your attitude, your
activity level, your health.

For instance, I smile at myself in the mirror each morning. Changes my whole
attitude. Doesn't matter how fake or forced; doing it and seeing it are a
strong enough effect to mold my mental health. Even if I don't mind being a
grouch myself, I do it because my family and friends benefit.

~~~
teddyh
The “placebo effect” does not mean what you think it means. All those things
you describe are real actual things and are not necessarily examples of the
placebo effect.

~~~
VLM
Correct but you could have given him an analogy to help him understand.

A non-placebo analogy of "Just going through the motions" would be stretching
and warmup before serious exercise, which provides a medically proven
advantage to overall systemic performance, lower likelihood of injury, faster
overall gains, etc.

Now a placebo of "Just going through the motions" would be skipping the
exercise and watching an exercise DVD or perhaps watching a professional
sports game on TV.

------
mathattack
My 2 cents on vaccines... Scientists are sometimes wrong. (Perhaps too
frequently in medical science) Scientists sometimes fabricate evidence. (And
when they do, they should go to jail, but that's another story) Usually the
system self corrects over time.

Bottom line - if I have to decide on vaccinating my kids based on the advice
of the general body of accepted scientific evidence, or a porn star, I'll go
with the scientists.

------
daemonk
There is always going to be signal loss from the raw data of the actual
experiment -> the scientists quantizing and interpreting the data ->
formatting and presenting salient parts of the data to get published ->
popular science writers condensing the publication down to a few paragraphs ->
general public interpreting pop science journalism -> word of mouth from your
friends who read a pop science piece.

It's the age old problem of going back to the primary source. What exactly is
the primary source? The raw data? The original publication? You can even go so
far and claim the original experiment wasn't done correctly and you don't
trust the experimenter. It's an infinite regression that just isn't very
practical.

The key I think is more openness in scientific publishing and data. No more
anonymous peer reviewers. Make negative data just as important as positive
results...etc

------
oleganza
On a free market, shamans and doctors invest their own time and money and can
lose their savings if their method does not work. In the long run only precise
scientific methods give reliable results while shamanism has random results.
People vote with their money for the most efficient and better proved way to
solve problems (provided kings, churches and governments do not interfere). Of
course, this iterative process does not give you a short-term guarantee that
you'll be healed, but neither does any sort of absolute authority.

We don't necessarily need a lot of people to understand a lot of complicated
things. We just need to not interfere when some people think they can solve
things better than before, even if you think they are mistaken. Doing
otherwise creates a larger scale hazard (cf. today's patent and regulatory
environment).

~~~
mcguire
In the long run, we're all dead.

The market can remain irrational longer than you can remain solvent.

There is an unlimited supply of wrong theories; if yours isn't working, pick a
new one.

~~~
dllthomas
_" In the long run, we're all dead."_

Speak for yourself, mortal. I'm taking this new immortality tonic...

------
Fomite
A particularly dangerous approach to "Where was it published": The original
Wakefield paper was published in _The Lancet_ , which is as good as it gets.
It was retracted by the journal, but that fact is occasionally omitted.

------
rubiquity
Unrelated to the article: I nearly jumped out of my seat when I saw a link to
Martin Sustrik's blog on here. I recently discovered his blog and really enjoy
his writing. He's one of the main programmers behind ZeroMQ and is now working
on a new messaging library named nanomsg, using some of the lessons learned
from building ZeroMQ. It's a great read if you enjoy the melding of technical
and social writing. Pieter Hintjen's blog is also very interesting[0] for the
same reasons.

0 - [http://hintjens.com/](http://hintjens.com/)

~~~
rumcajz
Thanks for kind words. I would like to point out the previous blog post (the
one about immortality) that I am particularly proud of :)

------
mcguire
There's an old quote from Larry Niven, I think, to the extent that, "If an
old, white-haired scientist tells you something is possible, he's probably
right; if he tells you something is not, he's probably wrong."

On the other hand, if a thousand old, white-haired scientists tell you
something, it may be either right or wrong, but it's probably the way you
should place your bets.

~~~
cousin_it
I don't understand that quote. Any scientist would tell you that perpetual
motion machines of the first or second kind are impossible, and they're
probably right. In general I don't know many examples of impossibility results
in math or physics that were later overturned. Anyone?

~~~
habitue
It's just a maxim, it doesn't apply in every case. I think the motivating
example is Einstein's fervent denial of entanglement in quantum physics. In
other words, he's saying the tendency of scientists as they get older is to be
more resistant to new ideas.

Whether that's true or not in general, is debatable. But it wasn't intended as
an ironclad law in the first place, just a tendency

------
ansible
With some basic science training, you should be able to read the paper itself,
see how they constructed the experiment, and have some idea if they did a good
job. And this is without knowing much about the specific field. Did they have
a big enough sample size, etc.

The press that reports on science could help with this too, but it is
unfortunate they seem mostly useless in helping to provide a critical eye.

------
pesenti
There are many more avenues for the non specialist to get a substantiated
opinion. In this case, not just doctors, but the general press, scientific
vulgarisation sites and blogs, facts surrounding the publication (all the
authors but one rejected the study) etc. It's the combination of all these
factors - these "evidences" \- and not just one, that should convince people.

------
bjhoops1
I found a bottle of Teething Tablets for my 10 month old labelled
"homeopathic." I threw them in the trash, pointing out to my wife that placebo
effect can only occur if the patient is old enough to comprehend that they are
receiving a treatment.

~~~
wrongc0ntinent
Didn't that make you feel a little better?

~~~
bjhoops1
Why yes. Yes it did. Maybe the real benefit of homeopathy is making people
like me feel superior... Hmm..

------
SeanLuke
Heaven forbid we should start basing our scientific judgement on the severe
numeric biases of Thompson Reuters.

------
wissler
When I point out the fact that science does not mean "just trust the experts,
they are experts", and the scientific method does not mean "peer review by the
anointed institutional authorities", the typical reaction is revealing.

The same people who bemoan a failing trust in our scientific institutions are
themselves furthering it, by adopting starkly authoritarian anti-science
attitudes. They compound the problem by rabidly attacking anyone for daring to
question, which is behavior more fitting to a religious dogma than to a
science.

 _" There are more things in heaven and earth, Horatio, Than are dreamt of in
your philosophy."_ \- Hamlet (1.5.167-8), Hamlet to Horatio

