
Roko’s Basilisk: The most terrifying thought experiment of all time - chermanowicz
http://www.slate.com/articles/technology/bitwise/2014/07/roko_s_basilisk_the_most_terrifying_thought_experiment_of_all_time.html
======
api
The average human IQ is 100. There are people with IQs upwards of 180, and
recent research has shown that the genes underlying this are likely to be
relatively common in the population. Given that IQ is extremely powerful and
is positively correlated with income, health, and numerous other positive
indicators, I think we must ask ourselves why the average IQ would be 100 and
not 180. Why wouldn't directional selection for IQ have just run this all the
way up to the max?

The only explanation I can see that makes much sense is that IQ must have or
correlate with non-obvious fitness disadvantages.

~~~
maxerickson
You left "By definition" off of your first sentence.

There's even been a substantial apparent rise in the measured intelligence of
the general population in recent history:

[https://en.wikipedia.org/wiki/Flynn_effect](https://en.wikipedia.org/wiki/Flynn_effect)

As to the idea monster, I'm not sure if it is a retcon or not, but I've seen
lots of people claiming the reaction was more if you believe that to be
dangerous why would you share it and not so much oh no what have you done.

~~~
api
I was being really cheeky above, but I do see this as a rather outlandish
example of how beings of very high intelligence seem prone to getting stuck in
odd thought loops.

I recall an episode of STTNG where they attacked the Borg collective in this
way: they gave it an idea for object that appeared geometrically valid but
could not exist in three dimensional space. The collective thought about it
and thought about it, devoting more and more resources to it, until it
basically crashed.

Personally I think the answer to the Basilisk riddle is computational
irreducibility:

[http://mathworld.wolfram.com/ComputationalIrreducibility.htm...](http://mathworld.wolfram.com/ComputationalIrreducibility.html)

Roko's Basilisk is impossible because to simulate the universe in such detail
would require something that _is_ the universe.

------
mindslight
So they took the Pascal's wager of their religion and reworked it in terms of
evil. Oops.

I think the usual response to worrying about being enslaved by the devil is to
invoke faith that Good will prevail.

------
JoeAltmaier
Or not. How about, other people work toward creating a benevolent AI, and I
just chill.

