I too am skeptical- except in extreme cases. Maybe if it's some sort of mindless bullshit site that I'm not really interested in I would leave it if it didn't load in, say, 5 seconds. But I don't really give a shit about the bullshit web. Life would be better without that anyway (except, of course, for companies who make a living selling ads on such sites).
But if somebody's leaving a page that has content they need because it doesn't load in 1 second, then I'd say they're a dumbass.
Is it? The only place I've ever seen push that view is Google. I've never seen any non-Google information reflecting that.