First: binaries that ship with the system are not included in this because they're platform binaries. Tools like ps or grep won't be included.
Second: once the binary has been checked for malware it won't be checked again.
Third: if the connection fails immediately the malware check is skipped immediately. So if you are offline or otherwise force the relevant server DNS names to be rejected immediately it shouldn't wait for any timeout.
Fourth: anything you build with the Xcode UI is automatically excluded since you are using the user interface and explicitly asking to run "unknown" code.
Fifth: Anything listed as a Developer Tool in Privacy will have its child processes excluded from GateKeeper scanning. This command will make the category show up and put Terminal in it:
"sudo spctl developer-mode enable-terminal"
Once it shows up there go to Privacy and check the box to enable Terminal.
If you run CI or automation you can do the same by putting your Jenkins or other binary in the list, causing all subprocesses to be excluded.
I was partly aware of many of these, and like I said I'm not 100% sure if this really is the culprit. Your comment is a solid reference on this topic though (better than most of what's on SO), but here's a few comments regarding my specific incident...
On your first point: Was aware of this, and this is the part that is most puzzling about the whole incident to me. I just checked my $PATH and I am indeed running the system binaries, so not sure how to explain this one.
On your second point: Was aware of this too, but I assume a lot of the slowness comes from JITted programs, for which there will be phoning-home for any new executable memory page (AFAIK), and the policy decision caching semantics for things that are not on disk are not as clear to me.
On your third point: While this is true for the connection that gets established to Apple's servers, I think this might have had to do with DNS being UDP based by default (AFAIK), so there is no explicit refusal, and it hangs on a timeout because of that, even though DNS can be done over TCP as well. Haven't investigated this though, just a hunch...
On your fourth and fifth points: I don't use XCode, but my terminal (Kitty) was already on the Developer Tool list when this happened, which makes the situation with ps and grep even more mysterious to me...
#3: Sorry I meant use hosts to skip DNS resolution and just map them to 127.0.0.1 directly (assuming you aren't running an https server locally). The names are ocsp.apple.com and api.apple-cloudkit.com.
Clearly something was trying to resolve hostnames but it may not be related to GateKeeper malware scanning. TBH a tarpit DNS server is not a case I have personally thought about before but is interesting to consider!
On your last point, would that include the jenkins-agent.jar file? If so, is there a way to dynamically add it? We download our agent jars on boot which I'm pretty sure would replace the preview permission, so it'd be great to add it after download.
The Developer Tool permission excludes a process's child processes from scanning, not the process itself, so you'd have to add whatever launches the Jenkins agent, not the agent JAR itself. That's probably either ssh or launchd; adding a blanket exception to either of those is probably undesirable (if macOS will let you do it at all).
Usually you have something configured as a LaunchDaemon or LaunchAgent. Put that process in the list so once it goes out and downloads the jar then executes it you're good.
I don't recommend putting java in the list. That would grant that power to all Java apps (or at least all that start through that copy of Java).
Second: once the binary has been checked for malware it won't be checked again.
Third: if the connection fails immediately the malware check is skipped immediately. So if you are offline or otherwise force the relevant server DNS names to be rejected immediately it shouldn't wait for any timeout.
Fourth: anything you build with the Xcode UI is automatically excluded since you are using the user interface and explicitly asking to run "unknown" code.
Fifth: Anything listed as a Developer Tool in Privacy will have its child processes excluded from GateKeeper scanning. This command will make the category show up and put Terminal in it:
"sudo spctl developer-mode enable-terminal"
Once it shows up there go to Privacy and check the box to enable Terminal.
If you run CI or automation you can do the same by putting your Jenkins or other binary in the list, causing all subprocesses to be excluded.