Not saying it was unfounded fear (I'm very pro space nuclear power), just pointing out what the actual contention is. People have an adverse reaction to anything with the word "nuclear" on it, even more so when it is put on top of a half million pounds of propellant and lit with a non-negligible failure rate.
The reality though is that failure of the rocket doesn't mean the reactor or fuel is vaporized and thrown to the wind. It's possible to construct the payload fairing to survive sudden, rapid disassembly and crash land intact off the Florida coast for recovery.
A nuclear-armed intercontinental ballistic missile is visually identical to a launch rocket carrying a peaceful payload to LEO. The neural association between "nuclear missile" and "rocket with nuclear reactor payload" is very strong.
It doesn't matter how many facts you throw at it. In the mind of the public, all launches have the possibility to result in another Apollo 1 or STS-51-L (Challenger) or STS-107 (Columbia). Every "nuke" is the Castle Bravo test. Every reactor is Chernobyl. And the public generally is terrible at risk assessment and actuarial math.
You could actually make the launch safer than an afternoon stroll on a Florida golf course, but no one is going to lie down in front of your golf cart screaming, "lightning strike!" or "angry gator!" or even "cardiac infarction!" or "terrorists!"
People may have no problem at all if you just call it "an NCG power plant" (for neutron-cascade generator) and say the details are classified for security reasons--that "security" being the type associated with a toddler's blanket.
The reality though is that failure of the rocket doesn't mean the reactor or fuel is vaporized and thrown to the wind. It's possible to construct the payload fairing to survive sudden, rapid disassembly and crash land intact off the Florida coast for recovery.