Selenium RC starts Firefox instances though which means I'll have to be running a X11 server on our CI box. I actually gave this a whirl with Xvfb but it still uses tons of memory.
It seems to me you really ought to be regression testing against the actual browsers your users will be using, unless you do absolutely no Javascript, in which case why run browsers at all? What situation are you in where you have somehow slipped between the cracks of this logic? (The answer may help guide other people's advice.)
"It takes a lot of memory" has a simple solution, after all: More memory. Recovering the cost of even $200 of RAM only requires finding one bug before a customer did!
