It's interesting because in Europe fishing has become an expensive pastime - that might prove the OP's point. When on vacation in Cape Cod I was pleased that fishing rights in sweetwater seem to be some kind of common property. In Europe it's state property that is licensed to private associations that sell things like daily permits which amount to around 15 to 30 euros.
I grew up in the northern end of Appalachia, with a lot of people that had less than an acre their doublewide sat on. Nothing stopped them using the ponds and lakes, the trail systems, or the technically private but defacto public access swathes of paper company land.
Not only did they use them for recreation, they often hunted for food. Technically illegal out of season, but national forest rangers tend to look the other way if you're hunting for sustenance, since those same poachers often would blow the whistle on more serious issues. (Illegal waste dumping, marijuana grows that abuse the forest with chemicals, meth labs that might explode and start fires, etc)
Plus as a practical matter, if someone takes one shot at a deer then moves regardless of success, it's pretty much impossible to pinpoint where the shot came from.
That's weird. A very large chunk of poor people I've met live in urban areas, and if they don't, often lack the time or resources to do those things. I must know a special type of poor person that OP hasn't bumped into.
Source: Grew up broke hillbilly