Just as an example: blind people can't drive at all, but they can walk miles pretty easily and safely. And visual impairments of various sorts are, IIRC, the number one disability among the elderly.
Slower reaction times are similar: they destroy the ability to drive, but they're not a huge deal for walking up a flight of stairs.
I don't see many people out in wheelchairs either, and I agree accessibility for them could (and should) be improved. I do see quite a few people with walkers, though.