Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I found this article quite compelling: http://selfawaresystems.files.wordpress.com/2008/01/ai_drive...

Imagine you do have an AI with a certain goal. It seems obvious that:

It will resist being switched off (because if switched off it cannot fulful its goal).

It will try, and likely succeed, in becoming more powerful (to better satisfy its goal).

It will not want to change its goal (because it sees that changing its goal means its current goal, i.e. the one currently motivating it, will go unfulfilled).



Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: