Furthermore, Seadragon employs animations to bridge perceptible performance gaps, e.g. loading new tiles. You get the blurry mess when zooming in as well, it gets slowly replaced by the sharper versions as new tiles get loaded. The use of the fading animation here helps mask the actual change, which makes it less visually jarring.
Another thing is that zooming uses a bit of inertia for the animation, with an easing effect. This takes much longer than Google Maps' or Leaflet's immediate zoom but (a) again, reduces the effect that the new frame upon interaction is totally different from the previous, and (b) during the animation new tiles are loaded already and replaced. This often means that by the time the zoom animation ends you already have the sharper tiles on your screen and the blurry ones were used predominantly during the animation during which you didn't notice the blurryness.
Panning also uses a spring to have some sort of easing for drags which means things are animated longer, thus more time for tiles to load without you noticing. Some people dislike that behaviour, I actually like it. It feels less "stiff" than what other tile-based viewers usually do.
In general, it's not blazingly fast, or at least not faster than other technologies. It just makes good use of animations and a few perceptual tricks to make it feel fast. Which is actually, what user-facing applications should concentrate much more, in my opinion. Raw performance matters much less than what users perceive (for interactive things anyway).