I'm not sure if labelling food as genetically modified is a useful warning, or if it would be the same as saying "wifi used in this building". WIFI isn't harmful, and eventually it's going to be everywhere and not avoidable. Yet you still have people who claim that electronic smog causes all sorts of things.
I'm happy to eat these things. I'm gently worried about releasing organisms into the wild with "exogenes" (or whatever they're called). Evolution is amazing and powerful and wonderful. And human intervention in eco-systems isn't filled with particularly great examples - a long list of invasive species comes to mind.
And I think I get your concern about the labeling as well. You're worried that people will assume the worst. I mean, you called it a "warning". But I expect it to be more like an organic labeling. Some people will care, most won't. Is it useful? Only to those who care.