I was really into SO about nine months after it started. It felt both exciting, functional and useful. I will be honest and say Karma certainly does and did motivate me. At that time also, I wanted to put together something like a "resume of answers". If a would-be employer wanted to see both my philosophy and my skill, they would just have to peruse my questions and answers.
So I "worked" answering SO questions for about a month, got to ~1800 Karma and stopped.
After that, I've mostly used SO to get questions answered. And I've noticed SO's quality has gradually gone down hill; I once could count on getting good answers to my questions and I just can't get answers to any substantial questions.
So what's going on?
When I began with SO, it was really big already but nowhere as big as it is now. I'd classify myself as a senior-software-engineer, generalist. I pride myself on not just solving problems quickly but solving them "well" - how to solve the problem at hand in a way that's maintainable, etc, etc. "Working" on SO, I would look on the new questions. Skipping the really elementary ones, I'd find fairly challenging ones that were within "the purview of a generalist". To answer the ones I knew, I had to type pretty fast. I still put a fair amount of effort into my answers. I wasn't the fastest but I was fast that I was getting Karma.
The thing is, it feels like now, to answer a reasonably answer-able question, I'd have to bang it out in second or be behind ten other answers and be ignored regardless of the quality. So I just don't bother.
So, problems I can see.
Scale in question-answers: Many people answering questions means an answer comes up quickly and that might be good enough for a lot of question-askers. But there's a tendency for the super-quick answers to be garbage-answers. They only answer the question at hand - the super-quick answerers have become kind of the Demand Media of programming problem solving.
Scale in question-askers: Many more people asking questions result in lower quality questions. There are lots and lots of questions that qualify as programming questions but just barely. And there are lots of people very quickly banging out the answers to these and getting lots of Karma for it.
Scale in low-signal-to-noise-ratio: With so much quality stuff, the skilled question answers and askers have left. The guy who knows c++ inside and out just doesn't have an incentive to race to answer ten trivial questions and so they go away not wanting to compete with the guy does race to answer all this.
So my diagnosis is scale.
Further, I think the SO have noticed some of these problems but their efforts to solve them have more or less failed.
A while back, there was a large rearrangement of Karma in which the Karma for asking questions was halved. I suspect this was because they noticed that a lot of questions were going unanswered (part-and-parcel of the problems I mentioned above). I don't think this "incentivization" process worked. Rather than incentivize people to answer unanswered questions, I think SO needs to "put the fun back" into answering questions. SO was once like I game I could "win". Now it feels like a game I can't win, that's at a scale where it's impossible to win.
The badges I get just don't cut it. I think Joel & Jeff again realize the "you can't win" quality of current SO and try to compensate with badges since I find "I've gotten a badge" each I've logged onto to my otherwise moribund account in the last eight months.
What I think you would need to produce a programmers community which could stay viable longer is to do things to make sure that people could succeed in the meat of the matter - answering questions. For example, you might force a delay in people's answers or a delay in answers of people below a certain karma. You might actually limit the number of answers a person could give in 24 hours to force people to improve the answer they did give. You might give people more karma for answering higher valued questions.
You might also limit the ability of totally random questions to jump to the front after a while.
You might adopt a variety of "quality measures" to discourage building karma with only simplest answers.
I'm sure my analysis is imperfect, subjective and based on my own predilections. Perhaps some people just love the questions and answers I'm calling "low quality".
However it may go, these are my "rough notes" for an improved Stack Overflow.
Just today, for example, I came across this question:
At first I wanted to ignore it because I didn't really have enough time, but then I remembered some CSS3 monstrosity I saw on HN once where they got really clever with border radii and I wanted to see if I could do something similar. An hour later, I had come up with a halfway workable solution to his problem and everyone comes out a winner. (If it's not clear, I'm treeface on SO too)
I typically don't care so much about karma...I care about being able to deeply understand the tools I'm using, and SO really forces me to see them from a huge number of angles.
Yeah, it's really hard to analyze the communities/applications that have sprung up on the Internet. A generality is a view of giant table - it's won't tell you everything but it's better than nothing. When an internet community gets seriously big, it can be highly functional and highly disfunctional at the same time.
Among other things, SO's current quality probably varies a lot depending on the language you use. I went from using Ruby to using Matlab and finally back to using c++ in the time I was writing about. That might have biased the sample I had of the infinite dimensional SO-space. SO/Ruby might be great still for all I know.
Also, the stated reason for lowering the karma for asking questions was that there were some people who were earning massive karma gains by just asking tons of duplicate and "easy" questions (stuff like just going through the ruby docs and asking what each function was for).
My vague, working hypothesis is that Karma gets people moving but habit keeps them going. The habit of rapid question question answering might carry the person well past what the incentive of karma does.
Also, the effect of having to answer quickly, before someone else does, directly reinforces quickness whereas a Karma limit is rather indirect.
Plus, each question answered can keep earning karma beyond the one day limit.
Of course I'm speculating in this whole process...
Wikipedia demonstrates that a cooperative model with minimal rules can work staggeringly well, even on a massive scale. There is such a strong intrinsic motivation to contribute. It's a terrible waste to subvert that shared interest in favor of a conflicting one.