Internally Second Life calls their countermeasure the "gray goo fence" but I can't find many good descriptions of how it actually works online, except some mentions that the restrictions on how much something can "rez" increase exponentially. This link was the best description of it I could find:
I believe that, since the attack mentioned therein, they've done other things, such as restricting certain scripting functions to trusted people. But folks have been known to find exploits anyhow.
Of course, feel free to correct me if you actually play Second Life. I've only been following news of it from the outside. I've never actually played it.
I've never understood why people are that worried about it with nanomachines, though. We already have gray goo (though much of it, like algae, is green). I mean, most single-celled organisms are capable of that kind of mass replication. And they can be found pretty much everywhere already. I'd be more worried about accidentally engineering a super-virus or whatever than gray goo per se.