1. SVG has all these seemingly useful features that have non-obvious gotchas. Example: you can give rendering hints for svg or any child shape nodes, like "crispEdges" so that straight lines are aligned to the pixel grid (plus it might turn off anti-aliasing, but that's another matter). Great-- I've got boxes with 1-pixel borders that should always be pixel aligned to look sharp, so I choose that.
Then I notice that when I drag one of my "crispEdges" boxes to the right with the mouse, I get this weird flickering due to the "crisping" algorithm aliasing the sides of the box to the pixel grid in a way that changes the width of the visible box. (I.e., if I move the box 1 pixel to the right, the left side might align to the grid that is greater than xpos, while the right side might choose the grid that is floor(xpos).) As a result it looks like the box is "crawling" across the screen, or up the screen if you drag it vertically.
So I went back to manually aligning those boxes to the pixel grid to guarantee consistent box size.
2. The "paint server" concept is an over-engineered mess. It's incompatible with CSS gradient spec which makes it a pain to figure out how to make CSS and future of SVG play well. Sane frameworks like Snap.svg abstract away that and anything else you'd likely find inside <defs> tags. (Also, someone here on HN mentioned that <defs> junk depends on the id to reference content, which easily creates nameclashes if you use multiple inline svgs.)
3. There's some churn in the DOM interface for SVG-related utils. For example, at some point I wanted to get access to normalized path instructions for a given SVG path. But Chrome recently deprecated the old way of doing that and is in the midst of replacing it with a new geometry interface.
4. The syntax for specifying an arc inside an SVG path is weird. Like, so weird that code in an SVG library I read obviously just went straight through the instructions listed in F.6.4/F.6.5 of the SVG spec (using the exact same variable names) to simply hand off the values to Cairo, and the library still had a bug somewhere. That one isn't really a big deal-- I'm just fascinated by how the spec writers favored endpoint parameterization consistency over the user's ability to draw a pie chart without having to ask questions on StackOverflow.
5. The complexity of the spec and its interaction with the other HTML5 specs is obviously a pain point for browser devs. Sometimes I use devTools to make a quick-and-dirty realtime codepen inside about:blank which just parses a contentEditable div and injects it as innerHTML of an SVG. It's nice because you can prototype things with patterns and gradients quite easily this way. Except you can't reference gradients/patterns/etc. in Firefox this way because of god-knows-what-conflict between about:blank's baseURL and SVG's funcIRI. (Although you can base64-encode the data and inject it as the src of an img under Firefox.)
You see things like that all over. For example, the burgeoning web animations API looks really nice and works with SVG. But it works on CSS properties and not SVG attributes.