I drive a lot, thanks. If you can prove me level-5 or even very good level 4 autonomous driving, and that a computational driver makes radically fewer fatal mistakes than a human, then I'm with you. In other words, if you can satisfy a good regulatory regime like the say, airplanes or drugs, then great.
The thing is that we're not going to get there if we disallow anything in between 2 and 5 just because it can be a danger when used incorrectly. Level 3 by definition[0] is where the driver can look at their phone until the car/beeps at taps them to start driving again, and we know that system won't be able to tell when it needs to request human intervention perfectly 100% of the time, yet we need to get level 3 systems before anything above it.
Even autopilot in its current form has proven to be such an attractive nuisance for abuse that there is a market for those stupid steering wheel weights. How can you possibly not appreciate this problem?
Short of that, it's a luxury and a danger.