The great majority of the recruitment for the study was done through social media associated with Freakonomics... in the recruiting for the study, I emphasized that I was only interested in people who were having a difficult time making a life decision... Finally, this is a group which is apparently attracted to the idea of using a coin toss to potentially resolve major life dilemmas.
Not that we should automatically dismiss the results, but that is a pretty important qualification.
Also:
Empirical economists are increasingly moving from a role of consumers of data to producers of data. This paper represents an extreme expression of that trend. It is difficult to imagine how one could hope to answer the questions addressed in this paper without generating the data. As the prominence of social media grows, opportunities to recruit subject pools for randomized field experiments from broad swaths of the population will only increase.
As cool as that is, you've got to worry that if social scientists have the ability to conduct significant studies as a result of their popularity, we'll start to see a world where the most prolific social scientists are necessarily more and more like entertainers.
Levitt's been more entertainer than social scientist for a while now, and as for other social scientists wanting a broad section of candidates for field experiments, social media associated with the university or other third party, or even paid social has the same recruitment potential without the systematic biases of Freakonomics fans
Also:
As cool as that is, you've got to worry that if social scientists have the ability to conduct significant studies as a result of their popularity, we'll start to see a world where the most prolific social scientists are necessarily more and more like entertainers.