Problem:
We will probably create AGI before we have a formal model of human ethics, which is not an urgent problem but a potentially catastrophic one.
Proposed solution:
We assume we want that AGI (Artificial general intelligence) to be aligned with all life and we want it to produce maximum "good". We then define "good" as whatever any unit of life defines as such. (Interactions between units will produce additional emergent definitions.) Then instead of optimizing for any of those definitions we set AGI's utility function as maximization of the definitions, that is: maximization of the quantity of both units of life and interactions between units, and then to make it all safe we forbid it from interacting with either. Maximizing the quantity of life without interacting with it means both shielding life from cataclysmic events and creating space for life's expansion into places where no life exists (outer space).
More details: https://pni.ai/future
My intuition is that this can be expressed formally, simulated and proven/disproven and if proven may lead to a future with potential for growth that is hard to imagine, yet without any risk to life and without making humans obsolete. There is also no benefit in being the first to create such AGI and all the reasons to cooperate as it produces universal "good".
Can you guys poke holes in this?
