What I mean is that PhantomJS engine still falls short of a real browser. Advantage of running a real browser with Xvfb is that you won't ever have to worry about if the page is going to render properly or have it behave the way you expect it to. The overhead is not of a big deal, in fact it's a tradeoff for rendering accuracy.
Don't get me wrong headless browser like PhantomJS is headed in the right direction but if compatibility with website is crucial for your success, it is not enough and I just don't have a lot of confidence that it will replace a real browser with Xvfb.
Maybe you'd be able to run more instances with PhantomJS compared to running Xvfb and a real browser because of lower memory consumption. I don't know if this is true.