Ah yes. I've heard this argument. It's pretty puerile.
Land, utilities and the environment aren't like human-made resources. They're finite. So there's some inevitable decision-making there. This may limit your absolute freedom, but no one promised you that anyways.
It seems to me like having a majority of residents and workers supporting an initiative for local civic planning has a lot more legitimate authority than, say, a national vote on the same subject. At some point, you have to accept being an outlier.