I don't believe you can in OS X - even though it's vector drawn, a lot of assets are raster.
Apple pushed (weakly) a "Resolution Independent" interface in 10.4 (I think) that would have promised drawing the screen at whatever DPI you want without looking terrible (and giving people who want larger text a better option than decreasing resolution)
It didn't really happen; though, and recently I remember seeing them push pixel-doubled assets, ostensibly in preparation for desktop retina displays in the future.
I didn't know GTK3 was vector drawn - that's really awesome!
Resolution independence in OS X didn't happen because of UX issues - if given a very small space (16x16 or 32x32), a hand crafted pixel icon will generally look better than a scaled down vector icon.
For examples, compare the sidebar Finder icons with their full size counterparts, especially in the Places subsection. Some are scaled down versions of the larger icons (Documents, for example), whereas some are stylistic extractions (Downloads becomes a green download arrow in a circle).
I'd hope that when 2x mode comes to OS X for high resolution displays, when you take a screenshot in 1x mode it would use the 2x mode assets to create a higher resolution screencap, but that does sound like a lot of work for a very minor feature that would increase screenshot size.
This is why Gtk+ implements icons as a set of vector / raster images to be displayed at various sizes. It makes resolution independence really easy. This has been important to Gtk and GNOME for a long time since it's a big issue for accessibility which is not only important socially and morally, but actually a key legal issue for adoption in large companies and governments.
I manage to take screenshots as pdf:
Unfortunately, this just gives me a pdf with rasterized graphics.