It’s probably not part of their repo, but exists alongside, possibly gitignored. Either way in the same workspace the IDE will have to invalidate a lot of it when you switch, and again so when switching back. Separate clones, or worktrees, mean they don’t get invalidated and your IDE just sees them as a separate project altogether, each with its own cache, regardless of where that cache exists.
