macOS does a lot of automatic tracking of things to try and reduce the impact of the security system. There's a system called "bookmarks" which lets apps have access to things they created even in sandbox-isolated locations, it might be related to that.
I think terminal users aren't really in-scope for macOS security.
So maybe because in the past I granted iTerm access to Desktop, it still has access to everything inside even after I've disabled it. I tried making a new file outside of iTerm just now, and iTerm can still read it, so it seems directory-level.
iTerm is third-party software like anything else. Wonder if it got an exemption. Also, TextEdit evidently has access to everything without asking, so it's not just a terminal thing. Idk what's happening exactly, but I don't trust this sandboxing.
I think terminal users aren't really in-scope for macOS security.