These people realize you can only do so many tasks so why not delegate the least value-added (what are you going to gain by earning 15 seconds) of the tasks to someone - or something - else? It's a no-stress, win-win to give yourself one less thing to worry about.
People who evaluate the situation for themselves and cross on their own terms
Those are not the kind of people for whom that extra brain power will do any good.
Yes, I wait for the green light. Because every single time I make a judgment call that it's safe despite the red light, I have a tiny increase in the likelihood that there might be an accident. (No, waiting for green alone is not good either. There's always a risk. Situational awareness is good. But walking on red carries a bit more risk than walking on green, all other things being equal)
And since I do cross many traffic lights every single day, that tiny risk compounds. And compounds. And compounds. And if only a single, tiny accident happens, the time I saved by "making a judgment call" for thousands of crossings is all gone.
But hey, if you think life passes me by, more power to you. Glad you get to feel like a rebellious risk taker, living at the edge, always aware.
If there's one thing that I've learned being a pedestrian in NYC for the last 10+ years: Traffic lights are very loosely correlated with driver behavior. In my experience, you're far better off making decisions based on observing cars and drivers than faith that a red light means anything to a cabbie in midtown during rush hour.
Trust me, even in NYC, cars are slightly more likely to drive on green than on red.
Or, in other words, a green light is a necessary but not sufficient precondition if you want to achieve minimal risk for your crossing.
Any Joe Schmo can jaywalk, but it takes a real free spirit to decide no cars are coming and run that red light.
On my first day exploring downtown, I crossed when no cars were coming (or even approaching from far away) and got dirty looks from the gaggle of Seattlites in front of me waiting for the signal to change. (It did finally change as I stepped onto the opposite sidewalk.) After a few weeks, I just stopped putting myself in the spotlight like that.
Now, when I go back to NYC, people seem irritated when I forget where I am and stand there waiting.
Going back to London / NYC, I always forget to just do it and walk across whenever the opportunity presents itself.
Seattle's jaywalking law makes it illegal to cross, even when not obstructing a vehicle, and the cops love to write tickets for it.
That's expected - after all, everyone think's they're above average (http://www.cbsnews.com/8301-205_162-57568186/everyone-thinks...), it's the phenomenon known as illusory superiority... "On a scale of one to 10, you probably think you're a seven. And you wouldn't be alone."
(Disclaimer: though granted, I'd hazard a guess that most HN readers are generally above average.)
SF is filled with "wait for the walk sign" type of folks, while NYC is filled with "cross on their own terms" types. If you ever see someone waiting for the walk signal in NYC, it's a good bet that they're a tourist.
Trying to glean any meaning from such behavior is a futile exercise.
I'm actually stunned that you've made the opposite observation. For me it hasn't been an occasional thing, it's a constant, everywhere-I-look-in-both-cities kind of pattern.
If 30 people are crossing when the cars have passed but the signal hasn't changed, surely some of them are crossing because the group is, not "on their own terms," as the OP would put it.
Or they are crossing an avenue above 14th street.
Not arguing the technicalities of crossing the street or why laws are made, just noting parallels between a decision that people make unconsciously every single day and its relation to life and the "big picture."