Also, above 100 Mph or so the aerodynamics change substantially. There is a clear cut-off point above you are definitely in 'no mistakes' territory, even ignoring a bit of cross wind as you come out from under an overpass get get you slammed into the guardrail or driven off the road.
Add to that ice, snow, rain, night conditions, glare, low sun and so on and pretty soon the driver (automated or not) is not the limiting factor but physics and getting reliable sensor inputs is.
Anything over 140 will probably not be practical for mainstream deployment, even in Germany where there are lots of skilled drivers and it isn't rare to see an old lady do 150 Km/h there is a sharp drop-off above that point reserved for those the possession of more money than brains. It simply isn't practical from a fuel consumption and safety point of view and even the safest cars are death traps at speeds beyond that.
See 'crashedexotics.com' for the end result of that route.
Certainly when you get up into the sort of speeds things become very tricky and unpractical. The problem right now though isn't really that we aren't all driving at 100+, but that we still have all of those straight as an arrow 55/65mph roads criss-crossing fly-over country in the US. Those may have made sense at some point, but modern cars can easily do 100mph on those roads with no particularly unusual danger. Pretty much everywhere in this country speed limits are artificially low though. Most country/wooded area roads don't make any sense in cars with ABS or traction control; my limiting factor on most Pennsylvanian roads is the danger of deer, not road geometry or car capabilities.
I think with the spread of autonomous driving technology we'll be able to see even more proper, and reasonably safe, speed from cars.