I would expect checks for Xcode to go through xcselect rather than a simple directory check. Installing the command line tools (sudo xcode-select --install) might actually be a better idea to test this.
I thought the same, but actually this method worked for me when I wanted the the Spotlight "Developer" option to show up (the CLT were already installed). I have the Developer panel under "privacy" as well, even if I never installed Xcode on my machine
