installing a ton of fonts up front takes a pretty significant amount of space, installing a subset for their language/preference or letting the user manage it makes it VERY easy to fingerprint users based on what fonts they download, and doing any kind of cross-origin long-term caching is a security nightmare as it lets you begin to map out where a user has been just based on what they download.
It solves a problem that people other than web designers care very little about, but costs me money and creates a slew of other problems...
Personally, I wish it was easy to just turn off!
You could compare it with http/2: If you do a survey, you won't find many people even knowing it. That doesn't mean it's useless to them.
Most people already have attractive, readable fonts installed on their computers, which are likely either sensible defaults, or configured for specific reasons (e.g. minimum size to deal with eyesight). Web pages that render as blank white space for awhile, or briefly tease me with text before blanking it out, give me a much more negative impression than ones that simply render in my chosen default fonts.
On 3G or shaky Wi-Fi, I've regularly given up on browser tabs because all I see is an empty page, even after half a minute of loading and when most images have finished downloading. (Maybe other browsers are better than Safari, but I won't switch just to see prettier fonts.)
I've been blocking web fonts for a while, and it feels like I have to whitelist one site out of three because it depends on icon fonts.