
Bad science persists because poor methods are rewarded - feelthepain
http://www.economist.com/news/science-and-technology/21707513-poor-scientific-methods-may-be-hereditary-why-bad-science-persists
======
ChicagoBoy11
The article reminded of me of Robert Axelrod's excellent "The Evolution of
Cooperation." Axelrod uses similar computer modeling to create a tournament to
help tease out how cooperative strategies emerge from primarily self-
interested behavior. It is one of the best things I have ever read in my life,
and I'd recommend that book to anyone.

The real issue with scientific publishing is that there is simply no penalty
for publishing shoddy research. I know several academics who made quite a big
name for themselves on research that was later partially or fully retracted.
No one cared about that; there was no real reputational damage done. To tackle
poor science, such "poor" scientific inquiry should be "punished" in some way.
Similarly, it is terrible for the advancement of science that only novel or
significant results get published -- there should be a way for researchers to
benefit from publishing well-designed research which simply did not yield
interesting results.

How to do that? I think some of Axelrod's tournament provides an answer. Like
in his examples, the individual incentives align to yield a pretty poor
outcome to the members of his population (he runs an iterated prisoner's
dilemma game). However, correctly setting up the iteration parameter's, slowly
a cooperative strategy becomes the evolutionarily stable strategy.

I can see how this could also be the case for academics. There is no law from
up above that dictates that "number of papers published" is the ultimate
metric of success. There is a culture, and processes, and institutions which
have led that to be a leading indicator of academic success. If there were
real motivation and impetus to change this, there is no reason to imagine that
other metrics (and processes) could emerge that would much more highly value
scientific integrity and thoroughness.

~~~
j2kun
> I know several academics who made quite a big name for themselves on
> research that was later partially or fully retracted. No one cared about
> that; there was no real reputational damage done.

cf. the "power pose"

~~~
YeGoblynQueenne
It's hard to find information on what the problem was with that one, so here's
a link to an article from Slate:

[http://www.slate.com/articles/health_and_science/science/201...](http://www.slate.com/articles/health_and_science/science/2016/01/amy_cuddy_s_power_pose_research_is_the_latest_example_of_scientific_overreach.html)

~~~
erichocean
Ironically, "failure to replicate" can also be spurious.

~~~
j2kun
IIRC there were on the order of ten attempted replications, all of which
failed.

------
chatwinra
"universities and funding agencies [should].. stop rewarding researchers who
publish copiously"

You have to look at the media as well. When was the last time you read an
article about a failed experiment, ie: an hypothesis was DISPROVED? There's
rare/no coverage of this yet it is an important aspect of science[1], and ties
into what the OP talks about.

If we're going to improve how science is done, this is an equally important
area to focus on because it's the current bias for positive results that plays
a part in driving labs to produce lots of papers.

1 - [https://www.elsevier.com/connect/scientists-we-want-your-
neg...](https://www.elsevier.com/connect/scientists-we-want-your-negative-
results-too)

~~~
6DM
We'll they do when there's common myths that get disproved. Like that guy who
cracked his knuckles on only one hand for nearly all of his life. But yeah I
get your point.

[http://www.foxnews.com/health/2014/03/13/health-myth-is-
crac...](http://www.foxnews.com/health/2014/03/13/health-myth-is-cracking-
your-knuckles-really-bad-for.html)

~~~
PJDK
Thats the only time when things have a general interest. Negative results are
very important to people directly involved in studying something but what
general interest (or value) is there in a story "chemical xyz123 does not work
as an effective catalyst for the reaction between abc123 and def678" for
anyone not directly researching those things?

~~~
6DM
There is no interest to the general public. But that doesn't mean scientists
can't carve out their own online community like Wikipedia and start to
categorize all of this information. It would for them probably be what
stackoverflow is to programmers.

------
taeric
I would say it is less that poor methods are rewarded, and more that the
investment in proper methods is expensive, and thus penalized.

That is, nobody is particular looking for people doing poor work and handing
out rewards. However, the "proper" methodology that we want necessitates time.
Something that is expensive and there are already plenty of things eating at
the budgets of work out there.

I do think this can be made better. But I see no reason to think it is just us
chiding people for not doing better.

~~~
hannob
> That is, nobody is particular looking for people doing poor work and handing
> out rewards.

I'd challenge that. If you look for people with high impact factors you are
asking for people doing poor science. If you only publish "significant"
results and refuse to publish negative results you are asking for bad science.
There are structural problems that lead to bad science, and often it's not
about costs.

~~~
afarrell
Given that plan to decimate the economy of America's 4th largest and most
racially diverse city (Houston, TX), I am quite keen to know if these issues
also affect climate science.

Don't get me wrong, I'm still pretty convinced that anthropogenic global
warming is happening. But if there is a change that it isn't and we can avoid
all of the murders and suicides that come with throwing large numbers of
people in one area out of work, we should investigate the possibility.

~~~
lmm
These issues affect all science. That's not an argument for ignoring science
when it's politically convenient. (At least not any more than you already were
based on the results. Whatever your all-in number for how often science is
right or wrong is, that number already incorporates all these issues).

If anything I would expect standards to be much higher in climate science than
less controversial areas. There are a lot of people who would stand to gain
(or, equivalently, avoid losing) huge amounts of money if the climate science
was wrong, they will be going over papers with a fine-tooth comb in a way that
doesn't happen to most papers.

~~~
afarrell
> ignoring science when it's politically convenient

And I'm not making an argument for it. I'm making the argument that "getting
this wrong is personally costly for a lot of people, therefore we should be
examining these results with a fine-toothed comb before basing massive social
change on them."

But you're right, there would already be monetary incentive to do that.

------
bootload
_"... his finding also suggested some of the papers were actually reporting
false positives, in other words noise that looked like data. He urged
researchers to boost the power of their studies by increasing the number of
subjects in their experiments."_

Conclusions based on low sample rates, should be seen as poor technique, an
indicator of potential bias and suggest a _lot more_ validation is required
before acceptance.

------
VikingCoder
...20 researchers try to study something. But only one of them gets
statistically significant results (p < 0.05), so they publish and the rest
don't.

[http://xkcd.com/882/](http://xkcd.com/882/)

If we don't address this, then the whole research publication process is on
fire.

~~~
throw_away_777
The easiest way to address this is to simply move the p value necessary for
significant results - instead of p < 0.05 use p < 0.001 or even p < 0.0000003
(3 x 10^-7), as is needed in particle physics. Though the 5 sigma threshold is
a bit ridiculous. Lowering the p-value threshold significantly increases the
cost of hunting for positives due to random chance, at the expense of needing
more sensitive experiments.

~~~
jjoonathan
Nope. It just means that you have to come up with a shittier null hypothesis.
Of course, it still has to get past peer review, but this can often be
accomplished by complication and obfuscation, which have the side benefit of
giving you plausible deniability in case you are discovered. Sufficiently
advanced cluelessness is indistinguishable from malice, and science, in its
present form, rewards them both.

------
SFJulie
Which is a mild way that our education is poor since rewarding status over
competence.

Would parents sending their kids to Ivy leagues accept a kid from downtown
going to public school being given a better recognition as theirs?

We let parents influence education, and let schools bend the «natural»
competition in order to satisfy influential parents.

This cannot happen without regulations to tie diploma to a job. Without
academy's and authorities. The first winner in the bending of the knowledge
competitions all other the world are the kids of teachers.

There is a worldwide corruption without bribe of teachers achieved by a social
pressure.

What could be done? Burst the educational bubble?

------
aisofteng
It is my personal experience that bad science persists not only because it is
rewarded but also because a large portion of scientists just don't do good
work.

------
rwallace
This is the one part of the article that is counterintuitive:

> Worryingly, poor methods still won—albeit more slowly. This was true in even
> the most punitive version of the model, in which labs received a penalty 100
> times the value of the original “pay-off” for a result that failed to
> replicate, and replication rates were high (half of all results were subject
> to replication efforts).

How can bad results still confer a net reward on their producers with a
penalty like that?

~~~
winstonewert
I went back and read the original description of the model, this is what I
think is going on:

The average performance of a bad lab is worse then a good lab. However, a bad
lab might get lucky and not have any of their false-positives subjected to
replication. As a consequence, the top-performing labs tends to be bad labs
that got lucky. The selection method heavily favors being the top performer,
and thus the poor but lucky lab tends to win out, which is why it takes over
the population.

This casts doubt for me on their model, since fitness proportionate selection
would probably have quite different results.

------
Lapis_01
There is another dimension to this problem; universities in developing
countries are pushed to match what their colleagues in developed nations are
publishing. In doing this professors where I am currently studying (Brazil)
foist off research on undergrads and masters students.

The system gets even more ludicrous when it comes to translation and
publication, but that is a tangential issue.

~~~
CSDude
Same here, quantitiy over quality is way more preferred. Trying to improve
some research that has no value just to increase publication count.

------
YeGoblynQueenne
>> Ultimately, therefore, the way to end the proliferation of bad science is
not to nag people to behave better, or even to encourage replication, but for
universities and funding agencies to stop rewarding researchers who publish
copiously over those who publish fewer, but perhaps higher-quality papers.

I don't see why being prolific should be punished. No, a better solution is to
reward research that is replicated more than research that is not yet so.

This would be much easier to do than punishing anyone. It's hard to find a
good reason to punish someone because they published something; after all,
even very thorough research has a chance to be wrong, and let's not get into
what is "correct" anyway. And it's a bad idea to discourage people from going
out and tyring things that have a very slim chance of working.

Rewarding replication might even result in some labs specialising in
replication studies and that would be a net benefit for everyone, especially
the researchers who would love to do replication but really don't have the
resources to do it.

~~~
mtriano
Yeah, but it's hard to get funding to do a rigorous replication of something
that doesn't have an immediate commercial justification (for example legal
liability or FDA requirements in drug trials). It's a big investment where the
best case scenario is that you can be a little more confident in prior
results. And if you can't replicate the results, it doesn't necessarily mean
the original finding was wrong, it could just mean the original scientists
were better than the replication scientists.

Also, among scientists who have invested the time to earn a Ph.D., there is a
culture of wanting to break new ground and push humanity forward. There would
have to be a strong incentive to motivate them to invest the time in
replicating research for which they won't get the glory.

You would have to change the way science is funded to make research
replication a required step in the modern scientific method.

------
projektir
The current model is just not efficient for science and I'm honestly
continually surprised that it manages to accomplish anything at all.

~~~
smaddox
Lying by omission seems to be widely accepted now as necessary and expected in
order achieve tenure, secure funding, etc. I find it abhorrent. If you go back
and read experimental papers from the 1800's, before science became an
industry, they are full of detailed methods (including where to purchase
items), pitfalls encountered along the way, and potential detractors from the
main result. Not so today. Today, it seems to be all about selling a result
that may or may not be significant.

~~~
projektir
It's a society-wide problem that is caused by competition and status seeking.
Pretty much everything is optimizing for "winning", whatever that means,
instead of doing things correctly, conscientiously, and carefully. The latter
will often make you "lose", and there's no system rewarding you for it besides
an internal one. Then there's nothing left for people to do but try to "win",
because resources are scarce. The more you optimize for "winning", the more
nasty your methods get, because there's always something you can sacrifice,
until nothing is left.

Another unseen side effect of this is that a lot of people see this, combined
with the low returns from science, and decide to go do something else, because
if they are going to put up with the garbage they can as well earn more money
doing it. And some among them then realize that they can have a greater effect
by generating billions of dollars and then using them on whatever they want,
instead of wasting time in a lab and asking for scraps from the government.

Untangling this requires a complete mentality shift.

~~~
internaut
> Untangling this requires a complete mentality shift.

I agree with this.

> It's a society-wide problem that is caused by competition and status
> seeking.

I don't agree with this.

I agree that competition and status seeking can lead to bad outcomes.

However this is wholly about how competition and status seeking are channeled.
They are instincts, they come naturally.

Clearly in the present order they are not channeled in ways that lead to
superior or interesting outcomes since mediocrity is everywhere.

~~~
projektir
> However this is wholly about how competition and status seeking are
> channeled.

You can't truly channel competition anywhere because it's always routed to
itself. Competition implies a provider of gifts and those who chase those
gifts. The provider will always use faulty proxies to determine who to provide
to, thus becoming disproportionately important to the process. Acquiring the
gifts is always disproportionately more important than anything else. Status
is one such gift, with no redeeming value.

Competition is always about the agent that is winning, and the provider that
arbitrarily bestows prizes. That is where the focus lies by definition - the
objective of a competitor is to eliminate all others, not to produce anything
of value. Science should be about the results, and should bow to no provider.

> They are instincts, they come naturally.

A lot of bad things are caused by instincts that come naturally. That's
actually specifically why I pointed those two out: they're natural, and
they're extremely dangerous and we should always be aware of them creeping up
and keep them in check.

Instead, we created whole institutions to worship them.

> Clearly in the present order they are not channeled in ways that lead to
> superior or interesting outcomes since mediocrity is everywhere.

The "mediocrity is everywhere" thinking is part of the problem, as it drives
everyone to worry about how "mediocre" they look and rush after the best thing
ever all the time, instead of calming down and doing the right things.
Publishing a failing study is mediocre. Helping people in need is mediocre.
Doing a little bit of fitness to keep yourself healthy is mediocre. Yet, that
is exactly what we need more of.

Mediocrity will always be everywhere because it's a mathematical fact - most
people are in the middle. So, yes, it's everywhere. That's tautological.

I think you were trying to say something else.

~~~
internaut
> Competition implies a provider of gifts and those who chase those gifts. The
> provider will always use faulty proxies to determine who to provide to, thus
> becoming disproportionately important to the process. Acquiring the gifts is
> always disproportionately more important than anything else.

I think we are on the same page with respect to the dangers.

I am reminded of the experiments in which rats are getting hits of dopamine.
What is it called? Behavioral conditioning? Skinner conditioning? Very simple,
but very powerful.

And this I think is the specter that haunts Science/Education. Sheer force of
collective habit maintains the status quo. There is lots of evidence for
'work' (even workaholicism) but paradoxically diminishing results. I am
reminded of those students who write down everything the lecturer says in
class but don't have the time to understand what the lecturer is trying to get
across.

> You can't truly channel competition anywhere because it's always routed to
> itself.

Here we sort of disagree. I agree that it is potentially an uncontrollable
positive feedback loop but I think competition (and also our sense of
fairness, demotism, voting, equality) can be channeled.

To use an analogy, these instincts we have are like Water or Electricity.
Dangerous but controllable to great effect. The goal of good governance is to
'traffic shape' these forces into productive causes.

Competition in the wild is simply Hobbesian violence, one man against the
other. It requires the halter of a market with its price system and exchanges
to be funneled productively. There are huge problems of course but we at least
know a halter actually exists.

What I am calling 'fairness' in its natural state is a powerful raw instinct
to enforce conformity and uniformity among the tribe members. There is an
obvious biological connection to these instincts, to our genetics, our genes
in their voting blocks trying to spread their influence. This is known in
biology as R selection, where the system is optimizing for volume (the overall
hypothesis is called R-k selection).

In my opinion there exists no 'halter' (as of yet, but I hold out hope) for
fairness. I do not believe a proper mechanism has yet been invented by
humankind. Yes humans build institutions to control it, like standardized
education, democracy, republics, but I am unconvinced as to their strength
against a sudden surge in the tide of popular feeling. Sooner or later, like
the rise of the Communists, brute strength wins out over elegant attempts to
moderate, basically mob rule followed shortly by blood rule with tyrants. Year
Zero. A Reset. It has happened countless times in history.

Democracy leads to entropy. Competition has the potential to take us to
Moloch.

[http://slatestarcodex.com/2014/07/30/meditations-on-
moloch/](http://slatestarcodex.com/2014/07/30/meditations-on-moloch/)

Basically you either need to choose your poison, or we need to come up with a
better system. I currently err on the side of Moloch, because it is a slower
death than the alternative.

> A lot of bad things are caused by instincts that come naturally. That's
> actually specifically why I pointed those two out: they're natural, and
> they're extremely dangerous and we should always be aware of them creeping
> up and keep them in check.

> Instead, we created whole institutions to worship them.

That's about the size of it, yes.

> The "mediocrity is everywhere" thinking is part of the problem, as it drives
> everyone to worry about how "mediocre" they look and rush after the best
> thing ever all the time, instead of calming down and doing the right things.
> Publishing a failing study is mediocre. Helping people in need is mediocre.
> Doing a little bit of fitness to keep yourself healthy is mediocre. Yet,
> that is exactly what we need more of.

> I think you were trying to say something else.

This is interesting, thank you. You're quite right that over-optimizing is a
threat (Moloch) but I shall reformulate my complaint here.

Mediocrity _is_ a threat (Entropy), _but_ , and I feel this is important, if
we have enough fields or subfields of exploration, avenues of research then
Mediocrity basically loses its meaning. The division of labour allows
everybody to potentially be a winner.

It does not eliminate average results but it disperses them over a large area,
which should mean net progress across the board is higher. Does that make
sense?

> Mediocrity will always be everywhere because it's a mathematical fact - most
> people are in the middle. So, yes, it's everywhere. That's tautological.

We all know smart people achieving mediocre results (via Molochian
competitions).

We all know not-as-smart people achieving, if not success in the sense of
being on the first page of a top ten Google search results, achieving success
in their local maximum.

In my opinion the second is in some sense smarter than the first because they
are better adapted to their niche, they have a better foundation on which to
ultimately progress.

I suppose I could summarize my feeling on the subject by saying I care more
about increasing the whole population's IQ (or other metric) by 1 point, than
about collecting the 'cream of the crop' and curating talent, because although
the second initially appears most impressive, the truth is that the whole
network benefits much more from the first.

~~~
projektir
I started responding to your post but after a while I found that we're not on
the same page on so many things that you just seem to be stating as fact that
I'm not sure if this is bridgeable. All of these things are worthy of a major
discussion of their own.

> Sooner or later, like the rise of the Communists, brute strength wins out
> over elegant attempts to moderate, basically mob rule followed shortly by
> blood rule with tyrants. Year Zero. A Reset. It has happened countless times
> in history.

> Democracy leads to entropy. Competition has the potential to take us to
> Moloch.

> Mediocrity is a threat (Entropy).

> We all know smart people achieving mediocre results (via Molochian
> competitions). We all know not-as-smart people achieving, if not success in
> the sense of being on the first page of a top ten Google search results,
> achieving success in their local maximum.

These are major statements that you can't really just assume to be true or
that the other person finds them true. And I don't agree with any of them. Not
to mention the lack of context on highly vague terms like entropy.

A lot of this smells like neoreactionarism.

~~~
internaut
> fter a while I found that we're not on the same page on so many things that
> you just seem to be stating as fact

I try to make clear what my priors, assumptions are. And which portions I
think are personal opinion.

I don't mind my assumptions being attacked, but I do mind that people "show
their work".

How many internet discussions have you had with people who did not clearly
delineate what their premise was?

That leads nowhere because the premise has a tendency to evolve (ever more
detailed refinement) when disagreement.

If you know what I think you're not shadow boxing with a set of stereotypes.

> These are major statements that you can't really just assume to be true or
> that the other person finds them true.

Of course not, but I'm not making an argument for them here. I'm writing an
internet comment, not a book.

> And I don't agree with any of them.

That is fine, although I don't really yet know your interpretation of what we
see.

> Not to mention the lack of context on highly vague terms like entropy.

Here I must take askance. Entropy is widely used across topic areas but it is
not a vague concept. Entropy is another word for randomness and disorder.

In the context of a society, you could say a Dark Age was less ordered than
the previous age when the Roman empire ran the show. Typical hallmarks of
anti-entropy effort include roads, aqueducts and cities. Ruined cities,
disintegrating roads and falling bridges are results of entropy. The social
and physical meanings of entropy are intertwined.

If that is not evident to you then you are correct to say our discussion is
not bridgeable. If there is one thing I find disagreeable it is a relativism
that posits all states are equally good. At that rate people will be flinging
their brains into bins.

> A lot of this smells like neoreactionarism.

And what if it is? You shall find everything interesting happens at the
peripheral. Conservatism and liberalism are after all, if nothing else,
definitely repeated routines of thought. Hard to take good observations there.
Of course if there is nothing to fix then there is no need to!

~~~
projektir
> How many internet discussions have you had with people who did not clearly
> delineate what their premise was?

Too many. And I often have the same problem since my premises are often far
from what is expected. I try to bridge it as I can, but limitations of the
human language and text space and all that.

> Of course not, but I'm not making an argument for them here. I'm writing an
> internet comment, not a book.

But it seems like you're making an argument /from/ them. Your definition of
mediocrity is different from mine (mine is defined as: a person who will by
others be considered mediocre in some context, which often resolves to a
person who has low-to-medium relative ability in a skill). I don't have a
definition of "entropy" to begin with, because it's not really something I
care about. Randomness and disorder is the official definition, yes. I can't
splice "randomness and disorder" onto reality. Reality seems to actually tend
in the opposite direction, at least on a planetary level (i.e., dust ->
planets -> creatures on planets). Mediocrity, given these definitions, seems
utterly orthogonal to entropy. A football player called "mediocre" by others
because he fumbled during a random game, or a person who is moderately OK at
tech support being considered "mediocre" because they're not great, to me do
not at all contribute to entropy. So I can only imagine your use of the word
"mediocre" is something else.

> In the context of a society, you could say a Dark Age was less ordered than
> the previous age when the Roman empire ran the show. Typical hallmarks of
> anti-entropy effort include roads, aqueducts and cities. Ruined cities,
> disintegrating roads and falling bridges are results of entropy. The social
> and physical meanings of entropy are intertwined.

My problem is that the definitions are too vague. As well as culturally
biased. Rome is more ordered than mud huts. But is Rome more or less ordered
than modern USA? Mongol clans? Soviet Russia? Czarist Russia? Maybe there are
different flavors of orderliness? I'm not really sure. Not to mention, if you
consider it on a global level, you have to sum up the results in some way, and
then determine whether the entire Earth is more or less ordered than some
other timeframe.

I think there's a difference between "everything is relative and all states
are equally good" and "everything is not relative but I'm not convinced on
what's more ordered and I don't trust your methods, either".

Nonetheless, I still fail to see the relevance of the football player being
called mediocre or the tech supporter not doing too great of a job to the fall
of Western civilization.

> We all know smart people achieving mediocre results (via Molochian
> competitions). We all know not-as-smart people achieving, if not success in
> the sense of being on the first page of a top ten Google search results,
> achieving success in their local maximum. In my opinion the second is in
> some sense smarter than the first because they are better adapted to their
> niche, they have a better foundation on which to ultimately progress.

From the earlier post. This to me sounds like an amalgamation of just-world
hypothesis and "nature is good". I don't correlate any specific outcomes to
one's "smartness" since I don't believe in either. I don't respect "niches"
too much since I don't believe in the latter. I disagree with both, and THAT
disagreement is fairly fundamental, it's very hard to find a bridge between
the "nature is bad and the world is not fair" and "nature is good and the
world is fair" groups, or even the other combinations.

> And what if it is? You shall find everything interesting happens at the
> peripheral. Conservatism and liberalism are after all, if nothing else,
> definitely repeated routines of thought. Hard to take good observations
> there. Of course if there is nothing to fix then there is no need to!

Sure, let's say all the interesting ideas are there, but there is still a
problem in assuming a specific vocabulary or axioms when talking to someone
who is not already familiar. I have my own framework that's not conservatism,
liberalism, or neoreactionism, but I try to keep it more or less under wraps
or contextualize parts of it, because otherwise discussion gets very
confusing. The whole line about communists and democracy and entropy and year
0 didn't make any sense to me until I considered neoreactionist thought. For
other people I think a large portion of the post is likely just gibberish.

~~~
internaut
> Your definition of mediocrity is different from mine (mine is defined as: a
> person who will by others be considered mediocre in some context, which
> often resolves to a person who has low-to-medium relative ability in a
> skill).

That's a definition I'd accept. Perhaps we're not alien species after all ;-)

I definitely see that mediocrity is context dependent. Nobody is a super-man
excelling in all. People like Sam Altman and Peter Thiel have rotten days as
do us all.

However it seems like there are _pools_ of mediocrity in society. It is not a
stand-alone individual phenomena. You don't find many Nobel Prize winners in
council estates in Britain. Or any kind of winner apart from the Lottery.

While nobody can excel at everything, we should be a bit suspicious when we
see a place where there is mediocrity in everything apart from alcoholism and
gambling addictions. That would suggest either extreme assortative mating
(highly unreasonable, in the time scale of centuries) or that a group of
people is being subdued systemically on the memetic level. What does one say
when we see an entire area code suffering from what seems like depression or
some existential crisis of purpose? The contrast with former glories is
particularly sharp in the North of England, but if you've ever been to a
similar region you get it. It is like a giant question mark in the landscape,
unexplained. We _should_ see a roughly random distribution of talents, there
would be elites, yes, but there would also be a long tail. I don't see much of
a long tail in present society in America or Europe, whereas I am convinced
there _was_.

These thoughts lead me to the belief that mediocrity is primarily caused by
societal structure. This suggests that talent is allowed to be exposed
sometimes and not others, which depending on how you look at it, is either a
very dystopian view of society and/or a potentially very hopeful one.

And of course we are seeing these same 'pools' arising in Science and
Education. Whatever the 'rot' is, it is surely spreading.

> But it seems like you're making an argument /from/ them.

I have a proselytizing streak in me. When I believe something is _so_ , then
others must _know_. However like yourself I am not a purist, I cherrypick my
way through various lines of reasoning and often am of two or more
contradictory minds on one topic. Currently I believe NRx's positions on
Western society and its politics (The Cathedral) are useful insights with
predictive power. I think also that every model has limits. The utility of
Conservatism and Liberalism has run its course in the West and now it is time
for something new or old again. In the Asian countries such as China this may
not be true.

> Randomness and disorder is the official definition, yes. I can't splice
> "randomness and disorder" onto reality.

> Reality seems to actually tend in the opposite direction, at least on a
> planetary level (i.e., dust -> planets -> creatures on planets).

Let us stick to centuries or thousands of years at most!

I think depending on the scale of resolution you look at physical reality,
you'll see different things, but this is another topic.

> My problem is that the definitions are too vague. As well as culturally
> biased. Rome is more ordered than mud huts. But is Rome more or less ordered
> than modern USA? Mongol clans? Soviet Russia? Czarist Russia?

I can easily imagine comparing them! There are absolutes, like average life
expectancy, and then confounding factors such as efficiency gains making less
things to measure. As a brute estimate, you could multiply the number of years
of life by the number of people the empire or system can support. That by
itself should give us a fair guess at complexity. Or we could look at density,
since average city size should correlate with complexity.

> Not to mention, if you consider it on a global level, you have to sum up the
> results in some way, and then determine whether the entire Earth is more or
> less ordered than some other timeframe

We already do this really. GDP. I realize all these metrics will have flaws,
but if you are consistent about using them you can obtain a fair idea of
trends and won't miss unusual events e.g. The Black Death. If GDP is negative
from 2017 - 2117, projektir will be far from shocked if armed with that
information he/she steps from the Time Machine and observes humankind.

> I think there's a difference between "everything is relative and all states
> are equally good" and "everything is not relative but I'm not convinced on
> what's more ordered and I don't trust your methods, either".

That is fine, my methods I just came up with a minute ago notwithstanding
there may be good objective metrics for orderliness.

> I still fail to see the relevance of the football player being called
> mediocre or the tech supporter not doing too great of a job to the fall of
> Western civilization.

Many functions in society are options or choices.

Items like the ability to travel over distances, to not be killed, to feed and
water oneself, to provide for a family, these are not really options. If
enough people can't accomplish those kinds of tasks then society does actually
collapse.

Fundamentally, are you saying that if the symptoms of a degeneration appear in
a society, that the people of that society just don't or can't be trusted to
recognize them?

I mean we have records, letters from Rome and Egypt, and they seem to indicate
a very acute sense of catastrophic decline.

> From the earlier post. This to me sounds like an amalgamation of just-world
> hypothesis and "nature is good". I don't correlate any specific outcomes to
> one's "smartness" since I don't believe in either. I don't respect "niches"
> too much since I don't believe in the latter. I disagree with both, and THAT
> disagreement is fairly fundamental, it's very hard to find a bridge between
> the "nature is bad and the world is not fair" and "nature is good and the
> world is fair" groups, or even the other combinations.

You don't believe some people's native intelligence is higher or lower than
others?

You don't believe societal niches exist?

Perhaps this is a straight disagreement but maybe I just don't understand what
you're saying.

> but there is still a problem in assuming a specific vocabulary or axioms
> when talking to someone who is not already familiar.

Usually, true, but not in these circles I find, after all you understood what
I was saying.

> The whole line about communists and democracy and entropy and year 0 didn't
> make any sense to me

Entropy I explained, Year Zero is a reference to the Khmer Rouge.

[https://en.wikipedia.org/wiki/Year_Zero_%28political_notion](https://en.wikipedia.org/wiki/Year_Zero_%28political_notion)

The only thing I can think to add here is that in NRx thought Democracy is
taken as being Communism Lite, like how there exists a Diet Coke soda.

> For other people I think a large portion of the post is likely just
> gibberish.

That is partly by design. Journalists have short attention spans and suffer
from buffer overflows. They then fall back on 'Fascists!', which not even they
wholeheartedly believe.

------
bsder
While I'd like to slag science for this, we also have a bit of counterexample
in other fields.

In art, we have studies that show that those who produce _more_ also produce
_better_. And it compounds. There is no reason to believe that at least some
of this isn't operant in science.

The real problem is the lack of positive incentives for negative results. And
I don't know how you fix that.

~~~
manarth

      we have studies that show that those who produce more also produce better.
    

This is true of people producing more of _the same thing_. An artist who makes
a lot of pottery doesn't increase their ability in painting, or piano-playing.

In other words, do _the same thing_ lots of times, and the results become
better and better. This sounds a lot like replication in science.

------
acscott
"Science" has now many definitions. My personal understanding of "science" is
this:
[https://www.youtube.com/watch?v=EYPapE-3FRw](https://www.youtube.com/watch?v=EYPapE-3FRw)

------
martincmartin
Would it help to move away from print journals, to online journals with
comments sections so the community can discuss any potential flaws?

Publishing the data seems like it would help, although might not be possible
where there are privacy concerns.

~~~
maus42
The active, open, free discussion on findings and theories has been regarded
as one of the big ideas of modern science. That's what journals were for.

However, for some reason I'm doubtful if the "online comments section" is the
most conductive for the behaviour we want to encourage, after witnessing how
that particular concept did manifest in the newspapers when they went online
(compared to the traditional opinion / to the editor page).

------
nonbel
>"Not only are dodgy methods that seem to produce results perpetuated because
those who publish prodigiously prosper—something that might easily have been
predicted. But worryingly, the process of replication, by which published
results are tested anew, is incapable of correcting the situation no matter
how rigorously it is pursued."

This isn't referring to science, and these studies aren't being done by
scientists. It is research, and these are researchers. I wish people would
stop dragging science through the mud over these problems.

------
erichocean
Is it just me, or is there a lot less "bad science" being done in chemistry
and (testable) physics?

Maybe one way to decide if a discipline is "scientific" in nature is how well
the scientific process actually works within that discipline?

And if we discover evidence that the scientific process doesn't work well in a
particular discipline (for whatever reason—doesn't matter why, actually),
maybe we should stop calling that discipline a "science" and come up with some
other descriptor?

~~~
mtriano
What evidence/data have you used to form your opinion in regards to hard
sciences? I don't necessarily disagree, but considering the subject matter, I
feel compelled to press you for a data-supported justification.

~~~
sevensor
I've seen copious bad papers in EE and MechE journals. I'm not qualified to
evaluate chem or physics research, but you'd think if any field would be hard
to fudge, it would be engineering. Turns out this is not the case. The bad
engineering papers I've seen tend to draw overly-broad conclusions while
failing to document the methods adequately.

------
feelthepain
BMC Psychology is trying out "results free" peer review - reviewers judge only
the method then decide whether to publish [http://www.the-
scientist.com/?articles.view/articleNo/47081/...](http://www.the-
scientist.com/?articles.view/articleNo/47081/title/Reviewing-Results-Free-
Manuscripts/)

~~~
projektfu
That would be great if it were a form of pre-registration, but since nobody
puts p-hacking in the method section, it's a little pointless.

------
SilasX
We can generalize: for all X, bad X persists because some incentive structure
makes it locally optimal for people to produce bad X.

~~~
Retra
Well, it's fundamentally easier to fail than it is to succeed. Especially when
you try to do something difficult.

------
jwatte
1 measure all the things

2 regress variables until p < 0.05

3 publication!

------
kayhi
Poor methods which is a result of inadequate data/descriptions being provided
by researchers to the publication (also an issue with publications not
requiring it).

In many of the hard sciences there is not a requirement to list the products
used such as chemical reagents or plasticware in a given experiment.

~~~
rspeer
The publications don't seem to _want_ adequate data and descriptions. They
want papers that have the traditional format and fit in 6-8 pages.

Every paragraph you spend on practical details is one less paragraph you can
spend on making a good pitch for the relevance of your work, and citing lots
of people in "Related Work" so that the reviewer's favorite people are in
there.

They don't want your code, because that would interfere with the supposedly
blind review process.

Occasionally, I've seen a submission process that includes a way to submit
supplemental data, but it's always extremely half-assed. There's no guidance
about what this supplemental data should be beyond "maybe an Excel file or a
zip file or something", nobody will ever see it, and if you include too much
data it probably gives you an error and maybe even crashes and makes you start
over.

------
max_
"A new scientific truth does not triumph by convincing its opponents and
making them see the light, but rather because its opponents eventually die,
and a new generation grows up that is familiar with it" -Max Planck

------
dnautics
similar to:

[https://www.theguardian.com/science/2016/sep/21/cut-
throat-a...](https://www.theguardian.com/science/2016/sep/21/cut-throat-
academia-leads-to-natural-selection-of-bad-science-claims-
study?CMP=share_btn_fb)

------
dschiptsov
Why, being a member of a popular rich sect is more comfortable and rewarding
than being a seeker after truth.

