The question is: why does it make economic sense to refuse a cheap labor force? If suddenly some aliens came to the US and said "we have figured out a way to get free food. have all the food you need here", would there be complaints about the number of farmers that are losing their jobs or unfairly competing against free products?
Theoretically, it seems to me that the positive aggregate effect on consumers is larger than the negative (and very regrettable) effect on the farmers. For this reason, wouldn't it make more sense to have the consumers compensate the farmers by helping them pay for their education, etc, so that they can find new jobs? This seems to me as the efficient way to think about immigration.
Where I come from (Mexico, of all places), I hear people sometimes complain about how computers shouldn't be allowed on certain government offices, since they will leave thousands of people without a job. I'd say that, if they can find a different job, then we can all be better off by finding cheaper production factors.
More conventionally, it is also used in support of free trade, even when a foreign country subsidizes their exports: it is functionally identical to productivity improvements in the importing nation.
Cheap to whom? Cheap to the employer does not mean cheap to society.
Note that we have "cheap" people, they're just in the wrong places.
And yes, I've done farm labor for a couple of seasons. My father worked seasons in different parts of the country.