Can't have that though, people could end up in very dangerous places. I'll just say that the government probably has secrets. Hiding spots. Research projects. Etc etc. If you think governments aren't using the property of élites to squirrel things like that away you're being a bit naive. The last thing we'd want is some kid stumbling over some irradiated or bio-infested wasteland that happens to look really cool. Or maybe just seeing something they shouldn't?
Biltmore seems enormous, and far more than any family needed for a living space. But now we're starting to hear about things the government has done there over the years, and you get the sense that we've likely not heard about it all. And that's just stuff from the Gilded Age. Technology has advanced considerably.
If we are dead set on having a right to roam, then at the very least, you can't be able to sue the government or any landowner for any negative impact that you may incur by roaming. Like coming home and infecting your family or your school or whatever. A landowner should be able to post a sign that says, "danger: stay out" or something, and then you choose whether to stay out. But the consequences of the choice have to be on you. (But only for government type stuff. I don't believe a landowner should be able to lay booby traps for instance. But if the government has something going there, like at Biltmore? Well, they said "stay out", and you didn't. So that should be on you.)
https://en.wikipedia.org/wiki/Freedom_to_roam