The big concern is that driving as a human skill is reactive and adaptive, whereas ML (and software in general) models are pre-baked. If something happens outside the car's model, it will react unpredictably, and strange circumstances can arise while driving. AGI, as based on human cognition, would have the capability to adapt to as-yet-unseen circumstances.
I would suggest visiting r/idiotsincars sometime. Humans are terrible at reacting to situation outside of their experience, and essentially do random things all the time.
If an AI system's default response is "come to a stop safely" then it's going to be way ahead of a lot of human "unexpected situation" handling in cars.
There are many situations where "come to a stop safely" is the worst possible thing you could do.
Yes, people are bad at driving, because they don't pay attention, panic, make mistakes etc. But ML models tend to freak out at slight variations on mundane circumstances; a cyclist crossing the road at just the right angle and the wrong colour of bike, that sort of thing. The thing self driving cars need to avoid is killing people in broad daylight for no discernable reason, and that seems like the kind of thing that you'd need a mind for. It's the same issue as with adversarial image manipulation to fool image recognition; if changing 3 pixels can turn a frog into a toaster, you aren't really "seeing" the frog at all in a symbolic way, and not seeing a road symbolically seems like a recipe for disaster.
> The thing self driving cars need to avoid is killing people in broad daylight for no discernable reason
This, I think, is the thing that people miss when they say "self-driving cars don't need to be perfect, they just need to be better than human-drivers, who aren't actually all that great".
From a public confidence perspective, it doesn't matter if a self-driving car crashes one tenth, one one-hundredth as often as human drivers; as soon as you see a self-driving car kill someone in a situation that a human driver obviously would have avoided (like in the adversarial image kind of scenario), you've totally destroyed any and all confidence in this car's driving ability, because "I would never, ever have crashed there."
