You end up in, essentially, an philosophical debate over what issues are most important - don't go there. Such debates cannot be resolved (in the same way that the halting problem cannot be resolved) because they rest on fundamental disagrements about axioms of morality.
You're saying that the issue of "who's segregating from whom?" is unsolvable? Probably. It's also irrelevant.
Looking purely at the effects of segregation should answer if it's a good idea. For an individual it's possible to benefit from segregation but as a group we're worse-off. Arbitrarily picking out second-class citizens and lowering their opportunities is a short-sighted idea even if in the near term that means that more resources can be used to advance your own interests. It doesn't matter if people are choosing to be disenfranchised, it matters that we're all hurt by it -- suicide is roundly discouraged for the same reasons.
