I grew up in Cornish. Most of us knew. Our plumber was his plumber etc etc. he was left alone, there’s a lot of interesting art in the community to see, his house not being one of them.
The locals may have left him alone, but let's say 1% of the US population read the book and 1% of them decided to go have a look at him. That's still a person a day for 100 years.
As for all these writers, I think he wanted to be left alone, so leave him alone.