I've seen it done before (once, actually, I used it for the visual effect (you wouldn't think it, but a set of nested parentheses look odd when it's in literature (rather than coding, for instance (though I think it's possible that I borrowed the idea from a computer science class)))).