It needs to go well every single day, and only needs to go very poorly once. Not to conflate LLMs with actual super intelligence, but for this (and many other reasons related to basic human dignity), this is not a technology that a responsible society should be attempting to build. We need our very own Butlerian Jihad
The book daemon explored an interesting concept. It explored the idea that an AI could dominate and cause problems, not through super-intelligence, but through simple mechanisms that already exist.
Like the executive who deleted all her emails -- humans giving tons of control and access, and being extremely compliant to digital systems is all it takes. Give agent control of bank and your social media, and it already has all the movie scripts and mobster movie themes to exploit and blackmail you effectively with very rudimentary methods (threats, coercion, blackmail, etc.).
Just spoofing a simple email with the account it gained access too at the Meta exec's email (had it hit an email with an attack prompt), could have been enough to initiate some kind of thing like this. For example, by emailing everyone at the company and in contacts with commands that would be caught by other bots. No super-intelligence needed, just a good prompt and some human negligence.
The fundamental understanding is vague. I bet half of the people don't understand what we're talking about. And 90% of the people don't understand how they think, I guess. Moreover 90% of people 80% of time probably don't really think logically but do routine in a rudimentary thinking mode.
A federal agent executes a man in broad daylight because while a gang of them are beating him for having the gall to try and help a woman who had fallen to the ground they finally noticed he was carrying a gun. Meanwhile the government tries to tell you that he was the aggressor despite the entire encounter having been filmed from multiple angles and which directly contradict the official explanation. The best Altman can muster is that they have gone "too far".
This stops short of smirking at the users who continue to pay because they are ignorant of just how shitty the software system they are using is. Until everything blows up with the inevitable data breach or data loss incident that is.
People who don't see the utility in an Alexa just see the listening device they have paid to place in their home and might be tempted to smugly imagine that they would never be so stupid. But consider, do you have own an Android or iPhone device? You know, the ones with geolocation services, camera, and microphone? Do you also keep it near you almost all the time? You can probably see where I am going with this.
