I find it depends on the podcast. Generally professionally produced radio is well paced and edited, chatty enthusiast podcasts on the other hand have, for me at least, a poor information density and I get why someone might listen to them sped up.
Some emotive content can get lost in the speed-up too, imho. A caring tone and cadence can turn into blunt sarcasm at x1.3 and above. I notice it interview shows like Fresh Air.