What is hard about self driving cars is we want our kids to be able to ride their bikes on the same streets those cars drive on. Limit self driving cars to just hiways with only other self driving cars and we could have had them 15 years ago and there would be near zero accidents. Dealing with human drivers (who might not be sober), kids, wild animals, and the like is hard. (I'm sure those in self driving cars have a much longer list of things things that are hard)
Autonomous cars are hard because children, and wildlife do not have transponders, and worse have a habit of jumping out from cover right into your path. Killing children is obviously bad. Even though who morally would argue that wildlife don't matter still do not like the damage hitting the wildlife causes to the car.