I have always heard sunny snow days are the worst, because you feel cold, but the sun is still hitting you, plus reflecting off the snow to hit you twice.
I agree that it very much depends on latitude and time of year. In most of Europe, overcast days usually don't present much of a problem for me, but I remember spending a day at the beach in Ecuador under very dense cloud cover and getting one of the worst burns I've ever had