For me it depends on altitude. If they're at the height of the average Cessna, then I'm cool with it. I don't want drones buzzing 100ft above my head though.
The numbers I saw were about 400 feet. The bigger issue is that they aren't manned and are presumably flying at a greater frequency than conventional planes. My concern would be that they fail and fly into the side of my house at full speed. I also don't trust Amazon not to be using these drones for surveillance.
This is one of those things where knowledge makes it bad. As a personal example, having coyotes in my literal backyard on a daily basis is not something I would want. That it happens pretty much every day without me realizing it, though, shows it really shouldn't be a concern of mine. Other than making sure I have protection for any backyard chickens...