So far as I know, the only way to get gcc (or any other compiler) initially onto a Mac is by accepting the entire Xcode package. I'm posting this here because you mentioned the 3GB download, which always makes me cringe. I'm curious to see if anyone knows a better way. (I've had this conversation online in a number of places in the last few years, but never received an alternative.)
I don't see any reason why you can't compilers. Here's the instructions for Clang  and LLVM .
You can't really do a lot without the system headers and libraries, though, which are installed via the Xcode Dev Tools.
1. Essentials (1.75GB)
2. System tools (56.8MB)
3. UNIX Dev Support (563.7MB)
4. Documentation (Zero KB - sic! More seriously, it's a download at next boot, hence zero KB of stuff is installed immediately from the disc. Why they don't report the size of the later download, I don't know.)
5. Mac OS X 10.4 Support (Zero KB - again, sic!)
By default, 1-4 are selected and 5 is deselected. But here's where it gets interesting: you can only deselect 2-5 manually. 1 is a must. Care to guess where Xcode lives?
Here's how Apple describes "Essentials":
> Installs Xcode, Interface Builder, Instruments, Dashcode, Quartz Composer, GCC 4.0.1, GCC 4.2.1, llvm-gcc 4.2.1, GDB, and other developer tools. Also installs the Mac OS X 10.5 and Mac OS X 10.6 SDKs. All content is placed inside a location chosen by the user (default is /Developer on the boot volume).
I feel like John Belushi in the old "All I wanted was a pony, but nooooooooo" skit.