I think that the window manager is a little lower-level than what you're thinking of. If you're running a Mac, you would already have whatever proprietary program Apple uses in OS/X to display windows and the desktop environment. This is my understanding at least.
The whatever proprietary program Apple uses in OS/X isn't cutting it for me. I spend too much time getting the windows how I want them and it's never quite right. I would like to automatically tessellate my windows as seen in the GOOMWWM. Is there a way to do that in OSX, and if so, what is this type of software called?
I suppose using X11 on OS X it must be possible to use another WM than the one provided by Apple (quartz-wm). After all, they're providing twm, so there must be a way to run it.
I think this is exactly correct. My understanding is that you can run a window manager like XMonad (and, in fact, I've met people who do) but then you will only be able to use X programs and not native Mac ones. If all you need is a terminal emulator, a browser and a text editor then this would work perfectly.
