Personally this has happened to me once. I was visiting my parents and the town I grew up in. I went for a walk by my old elementary school and walked onto the grounds. It was late in the afternoon, well after classes were done and most of the students and teachers would have left. A hundred yards away, some children were playing soccer. I stopped to watch for a bit. It was only a few seconds before a woman came up to me and very curtly asked, "what do you think you're doing here?" I said I used to go to school here, and I was on a walk. She said, "you need to leave." I asked her why and she repeated that I needed to leave. Just super hostile. I didn't want a big confrontation, so I left.