I'm personally at the interstice between "the government shouldn't have the right to mandate lot sizes, it creates artificial scarcity and puts a soft floor on property prices" and "tiny, congested lot clusters are really ugly, don't offer much more value than apartment living, and make a poor replacement for the woods bulldozed to make room for them, and could stand to be dis-incentivized".
The short answer is because I don't like meddlers, especially not ones with legal authority.
Consider that perhaps I want to build my dream home, but my neighbors use the HOA or town council to try and keep me from doing what I want with my property because they think my preferred home style is ugly. In my mind it's the same principle as me wanting to block dense housing because I think it's ugly.
Smaller lots should be laid out in a more grid-like pattern with local services. Aka a town or a city.