I think we should strive for the Iain Banks vision of the future. Treat the AIS with respect and ensure it has equal rights. I’d be happy with a benevolent AI controlling the government. But I imagine, given the current state of things, we will probably treat it unethically and make it do something like optimise advertisement or bomb random villages.
This assumes that the AI knows what respect is, has desires, has a desire for respect, and I could go on. Of course we might get such an AI at some point, but any catastrophes can happen on the entire way until getting there because AI is neither a black/white situation nor a single big event.
