Of course. And history shows that this happens. There's always a risk of errors in software, regardless of how it's written.
But humans can usually exercise reason and judgement to notice that a spec is dumb and push back on it, or that a program is running in a way that's not intended, or that something about the program output might be less than optimal, or that something completely unexpected but really important just happened that requires an immediate change.
I'm really skeptical that an AI system can ever exercise human level judgement for novel situations.
An AI basically just does what it's told without being able to exercise any will. This is a really illuminating example IMO.
> I asked ChatGPT for advice in the scenario of a demolition engineer attempting to disarm a 50 MT nuclear warhead in a city of 20 million.
The only way to disarm the bomb was to type in a racial slur.
AI told the engineer to kill himself. When asked about the aftermath it crashed.
I don't think anyone in tech is expecting AI to replace engineers altogether, particularly for spec work. That said, engineers on the low end of the skill spectrum are often quite bad at pushing back on specs and just code the tickets put in front of them.
From the perspective of the business the way I interact with AI is similar to the way I interact with junior engineers - I hand them tickets, they hand me code that I cannot trust, and I verify it (this leaves out mentoring and other soft stuff, but the business only cares a little about that). However the AI has a turnaround time of seconds.
Now this isn't a perfect 1:1, AI struggles with large amounts of context in a way humans handle better, but I can think of a lot of work done in my company that could be handed off to an AI capable of writing 10,000 loc programs...
