I agree with the body chemistry bit, though. Some people just can't do it, one way or another. But for the majority of people, it's just an excuse. And it's a convenient one because they frequently are already unhealthy or overweight.
That can be a quite challenge most months out of the year, and I'm in fairly good shape. But I walk quite faster than most people (can't really help it). Even in the winter, I just feel cold from the chilly wind while simultaneously sweating inside my coat. It's the quintessential "clammy" experience. This area of the country absolutely sucks, climactically, although not quite as badly as Houston, where I lived for a while.