
The acceptance of bad scientific practice to the benefit of productivity - gbear605
http://backreaction.blogspot.com/2018/09/science-has-problem-and-we-must-talk.html
======
raziel2701
J.M. Ziman wrote in 1962: "It is typical of modern physicists that they will
erect skyscrapers of theory upon the slender foundations of outrageously
simplified models."

I guess it is only a matter of time before the proverbial skyscraper
collapses. I have had first hand experience with publishing papers of
decreasing relevance. It's almost always due to professors being unwilling to
give up their mental model of reality. I had one professor tell me he would
not let me publish my result because it contradicted a paper he wrote in the
late 70s. It was the grossest violation of the spirit and purpose of science
I've encountered, and it quickly turned me off from the whole institution.
Egos in academia are often unchecked.

~~~
nerdponx
At least these physics models lead to testable predictions. You should see
what goes on in economics.

~~~
AlexCoventry
Curious about why the parent comment is getting downvoted.

------
ianbicking
"This problem cannot be solved by appointing non-experts to review panels –
that merely creates incentives for research that’s easy to comprehend."

That doesn't sound SO bad. Selection criteria seems like a major issue here,
and the author only talks about the validity of the results.

Why do we care about physics? Or psychology? There are answers to these simple
questions, but I don't think the answers inform the selection criteria. We
don't care about physics because academic researchers need to publish papers
or get funding.

Having any old non-expert on the panel could be problematic. Especially if
they become political tools, and right now they'd definitely become political
tools. But it would seem reasonable to me if there were adjacent experts. What
do engineers or chemists want to see in physics? These aren't hostile
interlopers. But if nobody outside a field cares about a line of research, or
trusts a line of research, then maybe that's a bad line of research.

~~~
addicted
You’re right. What we should also do is that all critical code that may
significantly impact lives and people’s wealth and security should be written
in a manner that can be understood by laymen.

So functional programming is right out. Anything more complex than BASIC is
probably not a solution. Probably only literate programming is allowed.

~~~
ianbicking
There was some short Ted Nelson video I watched a while ago (I'm afraid I'll
never find it) where he made this claim: they needed SQL and a separation of
concerns (i.e., DBA) to build up a financial system where it was hard to
embezzle money. And it's not JUST ACID and transactions that you need (though
you definitely need those things!) but also the ability to audit access to the
database-of-record, which SQL gives you.

Maybe the corollary is that you should build up an appropriate domain DSL so
that you can express your important claims or transactions in a comprehensible
way. It's not to say that you can't misimplement the DSL to your advantage,
but there's an implicit good-faith claim you are making that is different from
raw vertically-integrated code.

------
majos
A graduating PhD student in theoretical physics from a Top 5 school said
something similar to me last year. Roughly, he said it was very easy to slip
into bullshitty fooling yourself when doing theoretical physics research, and
it took vigilance _from within_ to do truly good research, because external
bodies were unlikely to really punish you as long as the math worked out.

~~~
dwaltrip
> it took vigilance from within to do truly good research

Well said. I've been mulling on the idea that a scientific mindset must
include a deeply embedded skepticism that isn't afraid to analyze and question
even the most pervasive and mundane aspects of one's own being.

Of course, one has to get through the day and take care of life, in a
practical fashion. But I think too often we assume our actions to be much more
intentional than they actually are.

------
majos
I like that this article includes suggested changes:

> For starters, every scientist should know how being part of a group can
> affect their opinion. Grants should not be awarded based on popularity.
> Researchers who leave fields of declining promise need encouragement, not
> punishment because their productivity may dwindle while they retrain. And we
> should generally require scientists to name both advantages and shortcomings
> of their hypotheses.

The first and last points look good to me. The middle two, not so much. They
certainly sound reasonable, but what does a grant system not based on
"popularity" mean? How is it different from what we have now, where we find
some (putative) experts in the area, put it in front of them, and ask what
they think? We need _some_ way of deciding what to fund and what not to fund,
and who exactly is better equipped to make those decisions than existing
researchers?

Similarly, I like the idea that people should be able to switch fields. But
the implicit argument seems to be that the emphasis on publications makes this
difficult, so publications should be de-emphasized in assessing researcher
quality. But what is better? Asking a researcher's peers? Then it is a
popularity contest again. Weighting publications differently, i.e. "what are
your 5 best papers?" rather than "how many papers do you have?"? Then who
judges the 5?

For all the criticism heaped upon bibliometrics: they get used because
assessing the quality of most research is difficult, so even noisy filters
like conferences (in CS) or journals are taken as a useful signal. It's not
clear to me what better options exist, especially when trying to choose
between many options. It's just not possible for one person or even a small
group of people to truly familiarize themselves with the bodies of work of all
100 candidates for a faculty position/grant/fellowship/etc.

~~~
rossdavidh
Agreed, and I would add that suggestions 1 and 4, although fine, will not be
enough to make a substantive difference. To be honest, Sen. Paul's suggestion
(if implemented properly, i.e. add experts from other fields rather than
political hacks) sounds more likely to help.

------
ccapo
I wholeheartedly agree with this assessment. I think the culture is extremely
toxic, and it pushes out many promising students and researchers.

------
TangoTrotFox
I think this article is looking at the effect rather than the cause. Why might
it be that so much of science has started to edge away from well... science.
Today nearly 1 in 8 people have a masters degree or higher. And in one in 33
have a doctorate or professional degree. [1] And these numbers seem to be
increasing. That's really insane if you think about it. We now have more
people with postgraduate degrees than had any degree in 1970, and postgraduate
education generally implies a necessity to publish. But discovering genuinely
new and novel information is not easy. So what do people publish instead?
Enter the current state of science and academia. I think the most informative
paragraph in this article was:

" _Or look at Brian Wansink, the Cornell Professor with the bottomless soup
bowl experiment. He recently drew unwanted attention to himself with a
blogpost in which he advised a student to try harder getting results out of
data because it “cost us a lot of time and our own money to collect.” Had
Wansink been aware that massaging data until it delivers is not sound
statistical procedure, he’d probably not have blogged about it._ "

Physics is one of the least susceptible to this issue, but no field is immune.
The issue is that 'science' has become more about publishing than actual
discovery. The two should be synonymous, but they are not.

[1] -
[https://en.wikipedia.org/wiki/Educational_attainment_in_the_...](https://en.wikipedia.org/wiki/Educational_attainment_in_the_United_States#General_attainment_of_degrees/diplomas)

~~~
throwawayjava
The statistics about masters and professional degrees are mostly irrelevant;
those aren't research degrees and, although there are some exceptions, people
with those degrees don't publish science.

1.77% of the 25+ population has a doctorate degree. Even among those, a huge
amount are not PhDs and are not really research-focused. E.g., go read the CV
of the typical University of Phoenix doctorate holder.

Furthermore, it's worth noting that the world got a lot more complex since the
mid-20th century. The entire field of Computer Science happened, for example.

~~~
TangoTrotFox
Definitely great points. On the other hand, even when you really lowball the
numbers - I do think the point remains. 1.77% is 3.5 million people with
doctorates in the USA alone. Perhaps we can see this issue manifest is
something more directly quantifiable. Postdoctoral researchers tend to be some
of the most highly educated and ostensibly skilled individuals in our
population. Yet an average postdoc [1] will tend to earn [much] less than a
mailman [2]. It seems difficult to explain this outside of a severe
oversupply. And should the rate of people pursuing doctorates be increasing,
which seems to be the case, I think we can expect things to get even more
silly both in terms of compensation as well as output quality.

[1] - [https://grants.nih.gov/grants/guide/notice-files/NOT-
OD-15-0...](https://grants.nih.gov/grants/guide/notice-files/NOT-
OD-15-048.html)

[2] -
[https://www.bls.gov/oes/current/oes435052.htm](https://www.bls.gov/oes/current/oes435052.htm)

~~~
throwawayjava
First, I'm generally skeptical that "the market" is accurately pricing the
value of work produced by either postdocs _or_ federal mail carriers.

Second, a postdoc is still a training position. Medical residents are also not
paid much better than mail carriers either.

Third, postdocs select for two types of people: those who want R1
professorships and those who need a position to stay in the country. Both of
those groups drive down wages. I.e., it's sort of meaningless to compare a
group of people who are working for the money to a group of people who are
working for a shot at a seat in the upper tier of a tournament-style
profession.

Fourth, both annual and lifetime earnings of PhD holders (at least in STEM)
continue to outperform high school graduates (including mail carriers).

There's an oversupply of people interested in R1 faculty positions, but that
doesn't mean there's a oversupply of phds per se. The labor market seems to
absorb these people quite readily and at apparently high salaries as soon as
they decide to quit academia.

~~~
TangoTrotFox
The market is not created by some third party. It simply represents the
meeting point between what people are willing to buy for and what people are
willing to sell for. That postdocs are willing to work for scraps relative to
their skill level is indicative that supply has gone way beyond demand. For
instance software developers also face similar issues of what you're
mentioning with lower wage foreign workers willing to accept much less thus
driving wages down, yet nonetheless high demand keeps wages high.

This is not particularly controversial and some have even suggested that
starting to increasingly restrict the number of graduate students admitted
could be the right path forward, as mentioned in this article [1] from John
Hopkins: _These and other suggestions for career preparation make one implicit
assumption: that the nation needs the current number of biomedical scientists.
A radically different approach to balancing the biomedical workforce equation
would be to simply reduce the “supply” — that is, admit fewer students into
science graduate programs._ The article ends on what was apparently supposed
to be an up beat note suggesting that instead of such draconian measures,
perhaps we can look to people who are doing things like aiming to go teach
high school science, with a doctorate.

~~~
throwawayjava
And on the other hand you see things like [https://cra.org/articles-
addressing-shortage-of-cs-professor...](https://cra.org/articles-addressing-
shortage-of-cs-professors-across-many-institutions/)

Just like there's a glut of wordpress plugin authors and a shortage of ML
experts, different fields of academia have different levels of supply/demand.

Also, USPS really is _not_ a prototypical example of free labor market: those
salaries/pensions/benefits will be a lot smaller if/when usps privatizes...

------
throwaway770
Is there any advice onto how one deals with research misconduct in the
industry? That allows to avoid expensive pitfalls like Theranos? Is leaving
the place the best option? Simply leaving the place doesn't seem optimal. It
feels like, it just leaves an opening for a recent H1 from a poor country,
with a different tolerance for dishonesty. And the end-result is continuing
misconduct.

~~~
raziel2701
I think you should speak up with the expectation that whistleblowers don't do
well, but with the knowledge that science corrects only if scientists correct
it. It's unlikely a single person can take down the culture of an industry,
but one person can certainly start a movement.

You're definitely well justified in feeling torn about this. I agree that
leaving in silence is not the best idea, but I certainly wouldn't blame you if
you did that. It's hard to take action.

