With nuclear power, you have a low probability of a very catastrophic event with consequences that aren't local. In a very simplified scenario, 50 countries can carelessly operate a nuclear power plant each, and only one of them blows up. However, the wind carries the fallout to another country that never used nuclear power. The people who get cancer, whose soil is irradiated, never saw any benefit from nuclear power, and they never had a chance to stop the plant being built, because it was outside their jurisdiction.
So please, could we finally stop comparing Black Swan-type events with non-local consequences to managable, local risks?
Also, the average coal power plant produces more radioactive waste than a nuclear power plant.
The real problem: people fail statistics. People can easily observe a nuclear plant when it shows up on the news, and assign a disproportionately high risk to it. The ongoing damage of coal plants doesn't make the news, because it falls under "day-to-day operation" rather than "disaster".
Even if coal plants cause more damage to the world, failures are local and manageable. When your nuclear plant fails catastrophically, as they have at Chernobyl and now Fukushima, you can't point to "but this is a six-sigma event!" as an excuse. Given this risk, fission plants have been neither economically nor environmentally preferable enough to displace other ways of generating electricity.