
Empiricism Is Not a Matter of Faith (2008) [pdf] - doppenhe
http://aclweb.org/anthology/J/J08/J08-3010.pdf
======
simonster
I've thought about this problem before. I'm a neuroscientist, and I would
often like to try out other people's modeling techniques on my data, but since
the vast majority of published papers do not have corresponding published
implementations, I have to implement the algorithms myself, and hope I'm doing
the same thing the paper did, and hope they did the same thing the paper says.
There is some pedagogical value to this, since it ensures a good understanding
of how the algorithms work, but often this pedagogical value is limited. In
many cases the main innovation in the paper is a better optimization algorithm
and not a different way of framing the problem.

There is little incentive to publish your code. Refusing to give away your
implementation does not in any way constrain your ability to publish, and
giving away your code has only minimal benefits for your career. On the other
hand, it's risky, since someone might find a bug in your implementation that
changes your results. Additionally, a competing lab might show their algorithm
is better than yours, or worse, improve your algorithm and publish a higher
impact paper based on it, which might affect the profile of your publication.
Finally, publishing your code means you have to package it in a way that is
usable by others. Given these facts, it is not entirely surprising that most
people would not publish their code.

Since science is pretty decentralized, it's hard to achieve the kind of large-
scale change in behavior you'd need to make code sharing standard. The only
people who could simply decide that people should publish their code and make
it happen as a consequence are the funding agencies (e.g. NSF), who have only
recently begun thinking about data sharing and have yet to make code sharing a
priority.

One thing I'd like to see happen is a "viral" AGPL-like license for scientific
code. Code would be freely redistributable, but if it is used in a commercial
product _or publication_ , the license should require that the modified work
be freely redistributed. Given the choice between building on publicly
available code or writing everything from scratch, I feel that most
researchers would choose the former, even if it meant releasing their code as
well.

~~~
Fomite
I think the big one is the lack of incentives.

Publishing your code costs time in terms of getting it polished for public
use, money by way of time. And it has very little benefit - you've lost
exclusive rights to a publication platform, and Github doesn't show up in
tenure portfolios.

~~~
glomph
In an ideal world the additional citations that public source code created
would counter some of these problems I guess.

~~~
Fomite
The existence of papers _about_ software is a decent step in the right
direction, but it needs:

a. People to cite what software they used way more heavily than they do (this
includes me). b. The effort to be rewarded in and of itself. A paper, cited or
not, is credit. There's a huge leap of faith that, after all the time prepping
software for release that anyone will use it.

------
glomph
Good article. Bad title for wider audiences though. I expected a faux
philosophy essay from a narrow minded empiricist. Instead I got an interesting
article about software publication within linguistics.

------
javajosh
All empiricists have faith that physical laws do not vary in time.

~~~
syncerr
Huh? A physical law is defined as a "... [conclusion] based on repeated
scientific experiments and observations over many years and which have become
accepted universally within the scientific community."[1] In other words --
something that has been shown not to vary.

Faith by definition a "belief that is not based on proof".[2]

So you're arguing against the validity of formal logic or the scientific
method or that a reached consensus cannot be trusted?

EDIT: I think you're overestimating the certainty that an empiricist would
have about future events.

___

[1]
[http://en.wikipedia.org/wiki/Physical_law](http://en.wikipedia.org/wiki/Physical_law)

[2]
[http://dictionary.reference.com/browse/faith](http://dictionary.reference.com/browse/faith)

~~~
pfortuny
Not that it _cannot be trusted_ but that believing it is a matter of _faith_.
This does not mean it is less human to believe, but that it is _patently_
human to have to believe something.

Do not confuse _formal_ logic (which deals with sintax, not meaning, so it has
no truth in itself) and the scientific method for which belief is utterly
necessary.

Trust is not a bad thing. But trust requires faith. Faith is not a bad thing.
"Consensus" is not an absolute source of unerring knowledge of reality.

~~~
leobelle
Here come the faith bearing apologists who take believing what you see as
faith, and then somehow twist that into making it ok to believe things you
cannot see. Because it's all faith.

When someone does this, you've encountered someone with a broken brain.
Someone incapable of critical thought and meaningful dialog.

Faith is not ok. Faith is a really bad way to come to conclusions. Faith is a
measure of last resort. If people who believe in the supernatural had anything
other than faith to cling to to validate their beliefs they would jump on it.
Instead all they have is faith. All hundreds of thousands of belief systems
out there with completely contradictory beliefs using the same flawed faith
mechanism to live their life.

You talk to Christian apologists, as an example, for any amount of time you
get all kinds of nonsensical rubbish that sounds like they're validating their
beliefs on eye-witness testimony from people long dead, to validate a belief
that heaps and mountains of empirical evidence you yourself can verify
contradict. It's like you wouldn't believe a living, breathing witness in
court if their testimony contradicted a DNA result, a single piece of
evidence. Yet happy feeling supernatural mumble jumbled about being able to
come back from the dead because some people thousands of years ago said they
saw someone come back from the dead is ok? No, that's them relying on faith.

Faith is bad. You don't use it to fix real world problems. And no heuristics
are not the same as faith, so don't go there.

Anyway, these people will use their keyboards and talk here about all kinds of
things, but you know, press them and their mental gymnastics will fall like a
house of cards, exposing their flawed mental states.

Empiricism doesn't require faith, and faith is a bad thing. Avoid faith
whenever possible. Use a heuristic, and verify when you have the time, make
sure it's sound.

~~~
andrewflnr
If you're _acting_ on the results of your heuristic, without being completely
sure, that neatly fits my definition of faith. This whole topic is arguably
one of mismatched definitions.

You're backlashing against other people's bad logic, and in so doing letting
it cloud your own thinking.

~~~
zAy0LfpBZLC8mAC
This topic has lots of problems with mismatched definitions, but there is an
actual problem hidden in all of that - namely, one side tends to use
inconsistent definitions, while the other does not.

Your definition of "faith" for example seems to be "anything you believe even
though you are not completely sure about it"? By that definition, some
fundamentalist's belief in god is not faith, but when some physicist tells you
that some house exists, say, then that is faith? Sounds like a pretty useless
definition to me - plus I doubt that that is the actual definition you use,
outside of providing a definition?

~~~
jerf
"Your definition of "faith" for example seems to be "anything you believe even
though you are not completely sure about it"?"

andrewflnr just told you what his definition of faith is, and that was
manifestly not it. Try reading again.

I notice this easily because I favor this as the most useful of the myriad
definitions of "faith" in real life. It isn't the only one, and it isn't the
most popular one, but it's the most _useful_ one. And that is the beginning
and the end of the definition; it simply does not encompass "what you believe
in even though you know it isn't true" or anything like that; it is an action-
based definition that pays little attention to what people say, which, in this
particular domain, is a major virtue. To see what someone has faith in _by
this definition_ (which, again, is not the only one), watch their actions.

~~~
zAy0LfpBZLC8mAC
No, he didn't, he only gave an example, and I extrapolated from that.

As for your definition, I don't think it's particularly useful, because there
already is a word for that, but well ...

@"what you believe in even though you know it isn't true": that strikes me as
nonsenical. How could anyone consider something to be true ("believe" it)
while knowing it isn't (thus presumably believing that it isn't true?)?

~~~
andrewflnr
Actually, jerf understood me perfectly.

~~~
zAy0LfpBZLC8mAC
Given that I obviously did not, then, you might notice that that is a somewhat
pointless reply.

------
renox
Bad tittle: a better one would be "Empiricism Should Not Be a Matter of
Faith", because our current 'scientific method' is very, very broken. I don't
exactly remember where (I think it was a project trying to reproduce results
by pharmaceutical research) but I remember that this project could only
reproduce results _half the time_ on papers which included 'double blind'
experiences, less than that when the paper didn't have 'double blind'
experiences! Think about it when you'll hear about the new 'discovery' :-( :-(

------
dcuthbertson
From a practical standpoint, the IPyton Notebook [1] seems to be a good
starting place to enable sharing of software and reproducing results.

[1] [http://ipython.org/notebook.html](http://ipython.org/notebook.html)

