
Timeless Decision Theory [pdf] - anacleto
http://intelligence.org/files/TDT.pdf
======
Eliezer
This is a relatively old manuscript, and I would recommend instead reading, or
at least first reading:

* [http://intelligence.org/files/ProblemClassDominance.pdf](http://intelligence.org/files/ProblemClassDominance.pdf)

* [http://arxiv.org/abs/1401.5577](http://arxiv.org/abs/1401.5577)

(The original TDT manuscript occurred via pleas for 'something, anything to be
published' that were satisfied by cleaning up an unfinished book draft and
publishing it as-was.)

~~~
JoshTriplett
Is there a canonical list of "remaining open problems" that need solving in
order to construct a useful AI (or rather, a more-powerful-than-human
optimization process), other than the construction and validation of a
sufficiently robust decision theory?

~~~
Eliezer
Producing this is my major current writing project. Meanwhile,
[http://intelligence.org/files/TechnicalAgenda.pdf](http://intelligence.org/files/TechnicalAgenda.pdf)

------
Moshe_Silnorin
I believe the folks at MIRI spend most their time on Wei Dai's Updateless
Decision Theory now.

MIRI is strange. Five years ago, I was convinced they were very misguided or
cranks, but I've been reading some of their logic stuff and much of it is
pretty neat. I also attended a talk by Yudkowsky at MIT and it was also very
good. I'm not sure what to think now.

~~~
Eliezer
I affirm that UDT has replaced TDT. Updateless Decision Theory is by Wei Dai
and Vladimir Nesov with major contributions from several others.

~~~
maaku
Why?

------
miguelrochefort
Let's say a person is faced with Newcomb's Problem every day, and its decision
is made public every time.

If the person has always taken B, then the Predictor will probably guess that
the person will take B again, and therefore place $1,000,000 in B. By always
choosing B, the person "wins".

If the person doesn't always take B, then the Predictor probably can't guess
what the person will take, and B will be empty. By taking A and B, the person
can only get $1,000.

The mechanism of Trust is exactly this. Whenever you're faced with a question,
lying (to your advantage) will always increase the odds of a better outcome
(like taking both A and B). However, that trick doesn't scale, and you can't
use it anymore once a lie is discovered, once trust is lost.

------
DennisP
Most of the argument I've seen for this involves some very theoretical and
unlikely circumstances. Can anybody boil down (a) the decision process that
leads to cooperating in the one-shot prisoners' dilemma, and (b) the reason an
agent would prefer to use that decision process? That would be a remarkable
result.

I confess I haven't read the whole 100-page paper, but I've scanned it and I'm
having trouble finding an algorithm concrete enough to be expressed in code.

~~~
Eliezer
[http://arxiv.org/abs/1401.5577](http://arxiv.org/abs/1401.5577)

------
aspirin
There is something wery odd about Newcomb’s Problem. If the Predictor can
predict my choice with 100% accuracy, it's not really a choice, is it? Free
will does not exist in this scenario, and therefore I cannot meaningfully
choose between one or both boxes. The question is nonsensical.

~~~
ThrustVectoring
Instead of choosing a box, submit the source code of a program that decides
whether or not to open a box. Then, the predictor reads the source code, puts
money in boxes, and only then does your source code run and open boxes.

Now, imagine that there was a computer program that makes the same choices you
do in the same situations, and the Predictor can read that source code before
filling the boxes.

That's the intuition pump I use for thinking about Newcomb-like problems.

~~~
Dn_Ab
I might be mistaken, but it seems to me that the predictor should also be able
to solve the halting problem. Additionally, if the entity to which the source-
code is a descriptor for is so simple that they can be predicted without full
simulation, can we really say they had free will?

~~~
Houshalter
The program isn't given infinite time to run so its not a halting problem
case.

------
comboy
[2010]

------
jonathankoren
Never mention
[http://rationalwiki.org/wiki/Roko's_basilisk](http://rationalwiki.org/wiki/Roko's_basilisk)

