It's not easy to make a decent living of a tiny farm, but some people have done it. Most of them target higher-end products and direct-to-customer sales. For example, Eliot Coleman had a 5-acre farm on the coast of Maine that apparently did fairly well selling fresh local vegetables during the fall and spring for many years. His book is excellent and fascinating: https://www.amazon.com/Four-Season-Harvest-Organic-Vegetable... One of his big staple crops was really good carrots during fall months, because frost makes carrots sweeter, and if you don't use machines to harvest them, it's possible to grow tastier varieties.
There are also some very tiny Vermont farms that produce gourmet cheese. This obviously adds a huge amount of value to their milk production.
So in general, you're right. But it actually possible to make a living selling luxury products to foodies. The stereotypical farmer in this market is somebody who retired young from Wall Street, and who has fairly impressive business and marketing skills.
If you buy from 1 farmer and then sell wholesale to 1 customer your not going to make significant money. If you buy from 1 farmer and then sell at a stall to 1,000 people you can make meaningful profit.