A good podcast app may help. For slower podcasts, instead of speeding it up - which may chipmunk or otherwise cause artifacts - podcast addict trims silence. You may miss stuff below the gate threshold, but it is still slick.
I only listen to 2 podcasts, I don't really and have never really liked the format. I prefer reading with music or silence, and have always loved weblogs for that. However, it's worth a shot to find a podcast app that "clicks" with your style of input.
With podcasting 2.0, the new features should allow producers that don't want to be part of an ecosystem to still make some income, plus you get streaming album art where supported, and liner notes and all sorts of other goodies.
I think podcastindex has a list of compatible 2.0 podcasting apps, but I'm unable to check.
