This is different. This is a horse you ask nicely to take you there but once you're locked inside it can take you anywhere. The issue is not safety, it's control.
Horses have control issues as well. And a properly designed autonomous car is likely to have a manual override or emergency stop, things for which there are no good equivalents on a horse.