Systems in the real world exist in competition. The only way for one AI system to take over would be for it to overcome not just all humans but all rival AI systems. For one system to beat all other systems, it would need to be smarter than all of them put together. This means it would need over half the computing power of the entire world. Amassing physical resources is relatively slow and every AI would be trying to prevent one AI from taking over, so gaining 51% of compute would be hard. Algorithmic improvements are impossible (he actually says this), so there's no way one AI could suddenly jump in capability. Compute will be distributed in a power law (for some reason). Every AI will serve as a check on every other.
You may notice this is all nonsense. Somehow it's impossible to amass power without being smarter than everyone else put together despite humans doing it all the time. Does Hotz think Stalin had a brain the size of a skyscraper that he ordered airbrushed out of all photographs? Somehow Hotz can confidently predict how future technologies will work despite us having no idea what they'll look like even in principle. Somehow having gods fighting amongst themselves will work out just fine for the humans in the crossfire.
This "debate" is just people summarizing science fiction at each other. The score is one Atlas Shrugged and nine Star Wars to three Terminators and one I Have No Mouth and I Must Scream.
You may notice this is all nonsense. Somehow it's impossible to amass power without being smarter than everyone else put together despite humans doing it all the time. Does Hotz think Stalin had a brain the size of a skyscraper that he ordered airbrushed out of all photographs? Somehow Hotz can confidently predict how future technologies will work despite us having no idea what they'll look like even in principle. Somehow having gods fighting amongst themselves will work out just fine for the humans in the crossfire.
This "debate" is just people summarizing science fiction at each other. The score is one Atlas Shrugged and nine Star Wars to three Terminators and one I Have No Mouth and I Must Scream.