Unfortunately, in the US, we've allowed our representatives to enact a hard cap on the number of representatives. So the representative you elect decides these thing for an increasingly larger number of citizens as population increases. So while you have a say, the weight of that say on choosing a representative is ever decreasing.
Gerrymander strategy dictates that you want as few urban districts as possible to maintain rural power. Raising the caps would weaken the benefit of packing opposition voters into fewer districts.
Yes removing the limit put on in the early 20th century would have broad ranging consequences, most of them would be to the detrement of the existing representatives.
