I guess ultimately, I see:
Fear = lobbying power + profit motive + self replication + the fact we all share the same ecosystem + potential for imperfect science
Of these, it isn't the science that worries me most, it's the motives of the creators and the politics surrounding it.
Finally, labelling items as containing GM appears difficult, but it's also a fundamentally democratic act. If the only way you can persuade people to eat something is when they don't know what it is, then you have a problem. I feel for the executive who absolutely knows his product is ok to eat. But if people don't want it, that's the way it is, and I don't think the answer is to solve the science only ... it's to reduce the equation above.
Edit: to be more precise than maybe I managed above. My problem isn't one of science. It's one of trust.
I'd avoid food labelled with GM for two reasons:
a) I don't trust the creators or the politics.
b) Not buying it shapes the world in a direction that moves away from GM.
This choice is removed if I can't see it on the label.