Here is the link:
Years after pondering this article, I felt I had figured it out: psychology is secretly the study of particular subset of 18 to 21 year old American women. The ones who study psychology because it's a female dominated study and all those psych students need their credit and 'participating' in research is part of it. Most of them are American because psychology is a bigger thing in the US than in Europe, or at least it seemed to be regarding well-known theories, so I presume most research happens there.
There is another big group. A lot of dead mice (neuroscience).
"After the tragedies caused by the use of thalidomide in pregnant women, the FDA issued “General Considerations for the Clinical Evaluation of Drugs” in 1977. This guidance document stated that women of child-bearing potential should be excluded from Phase 1 and early Phase 2 research, except if these studies were being conducted to test a drug for a life-threatening illness. If a drug appeared to have a favorable risk-benefit assessment, women could then be included in later Phase 2 and Phase 3 trials if animal teratogenicity and fertility studies were finished...In 1993, FDA reversed the 1977 guidance with another guidance document entitled Guidelines for the Study and Evaluation of Gender Differences in the Clinical Evaluation of Drugs."
"In a study that evaluated the inclusion and analysis of sex in the results of federally-funded randomized clinical trials in nine major medical journals in 2009, researchers found most studies that were not sex-specific had an average enrollment of 37% women."
From "Women’s involvement in clinical trials: historical perspective and future implications" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4800017/
They could be attractive to the desperately impoverished single adults, who are mostly men.
Stats on poverty by sex and age in the United States: https://www.statista.com/statistics/233154/us-poverty-rate-b...
Worldwide, the gap is even wider: https://www.washingtonpost.com/news/worldviews/wp/2018/02/14...
I should have been clearer than simply stating _desperately_ poor; I intended to convey that they are in a state of poverty where there are few other options available.
I wonder why that is, I would've thought poverty and homelessness goes hand in hand.
Maybe it's more to do with mental health? Of all the homeless people I knew, a sizeable portion of them suffered from serious mental illnesses.
I don't know, I just found it interesting.
It's a really horrible place to be.
Men are the disposable sex.
And the world stats aren't relevant as we're talking about psych studies that mostly take place in the US.
social psychology experiments use whoever they can get, often for class credit only
There are many analogs of that in Computer science too. For example in computer vision, CIFAR-10 is a de facto standard for measuring performance. A set of 60k 32x32 images. Good results on that data set doesn't necessarily translate well into real-world performance. But what can you do? Gathering huge data sets and having humans annotate them is incredibly expensive.
Another ml example is music recognition. There are several state-of-the-art methods for detecting notes in music. So one Chinese researcher tried to apply them for Jingju music https://www.youtube.com/watch?v=NJsTl342RhI. The results were... less than stellar.
> subset of 18 to 21 year old American women.
This is perfect!
I had a similar epiphany about linguistics, much of which appears to be the study of example sentences that linguists come up with.
Since you could invariably tell when something was an example sentence, this was obviously a different language from what people actually used. (Of course there are linguists who go out and study real language-use, but some actively dismiss this as mostly irrelevant "performance")
" [...] This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%."
98% replicability!! Compare that with psychology.
The first problem is that asking people how they speak or what is acceptable does not accurately represent how they actually speak when not observed/asked:
The second problem is that, even if the first isn't an issue, the most this can demonstrate is that linguistic example sentences are a subset of actual language.
I'm not sure about the amount of research but the amount of psychology students here in the Netherlands is huge. And it seems to function in much the same way. Often first-years get some form of credit for mandatory participation in experiments for higher-year students and sometimes regular faculty members. I'm also aware of some researchers going out of their way to get a different sample from the population but in the end it's a lot easier to get participants when you make them.
I don't know any easy way out here. Just because "folk psychology" can say formal psychology is too simple doesn't mean folk psychology and anecdote are more useful.
Wait, does participating in a study as a subject count for credit, or does working on a study count?
Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?
Some overlap does exist, but it’s significantly less than suggested by the requirement.
It was fun, but it was seriously disillusioning to see how absurd most psychology studies are. Two researchers talk to me for 5 minutes about women performing better than men on certain forms of math tests. They then sit me in a room alone for about 10 minutes presumably to try to make me nervous. Finally they give me a test, overtly handing me it on a pink sheet of paper instead of a white one. Then after the test they continually tried to get me to agree that I felt like I probably didn't do so well on the test. I refused - since I was certain I nailed it (it was a cookie cutter pre-sat math type test). They told me I'd receive my results by mail within 2 months. I never did. E-mails asking for such were ignored. The research got published and was of course some trite gender bias explains all difference in math paper.
At the time I just found the whole thing quite absurd. In hindsight, I think it was even worse. That study was prepped and ready for p-hacking. They were measuring and possibly varying a large number of variables there. There was the paper color, the isolation, the 'chat(s)', different sections of the test, were men/women over/under estimating their performance, etc. And they were actively trying to push people into agreeing with what they wanted to find. You're going to be able to find some sort of statistically significant correlation there, even if none actually exists.
> Wait, does participating in a study as a subject count for credit, or does working on a study count?
There was this course called "measurements and diagnostics" and for that you had to pass the theory/tutorials of the course and also 10 hours of being a participant in research. If you didn't do the 10 hours of participation, you could do them at any moment you want and if you completed it, you immediately passed the course.
My university has about 400 psychology students, so that is 4000 hours worth of people participating per year. And I'm pretty sure that more universities are doing this. In The Netherlands alone I estimate there to be about 40000 psychology students per year and probably about 5 hours of compulsory 'participation' in experimental studies. So a tiny country like The Netherlands is skewing this particular bias about 20000 participant hours per year.
Participating in experimental studies is quite fun (pro tip: go for the fMRI and EEG studies ASAP! ;-) ), but I didn't appreciate how I was forced to do it. I didn't learn too much from those experiments and I felt used. It furthermore is a way to be lazy regarding participant recruitment. I'm pretty sure that some startup could disrupt this particular problem. If you have ideas about it, email me, I'm willing to brainstorm and help you to look for ways to solve this problem.
> Aren't studies explicit about the demographics of their participants? And don't studies that make generalized claims usually control for things like gender and age?
Studies are explicit about the demographics, which is why the WEIRD article could be written. And yes they do control for things like gender and age and in general they are quite careful making those claims. The issue is though as a research field psychology isn't the study of the human mind and human behavior but of WEIRD people their mind and WEIRD people their behavior.
Like a lot of societal issues that are in popular media today, this is a systemic issue. Though I could imagine psychology researchers being like: "Hmm, I could get a representative group or I could just get some psychology students and churn my next paper out twice as fast!"
Here's another interesting phenomenon: a certain personality type (of which I may or may not count my self among) will deliberately supply false data which attempts to invalidate what we perceive as your study goal when forced to participate against our wills in this way. Its a small rebellion against an academic system that doesn't respect us or intellectual integrity, but is focused on pumping out shoddy research on unwilling participants.
Turn academics into a game and thats how it'll be treated.
I also have the choice of not working or wiring some other job.
I would say it's a responsibility not supporting forced on people against their will. Two very different things.
Voluntary informed consent generally requires that participation is completely voluntary, that any rewards for participating don't cause undue influence on the participant's decision to participate and that there will be no consequences or loss of benefits for not participating.
I would argue that to reach this standard the requirement to participate in studies to complete your degree would need to be known before enrollment (likely the case) and the actual study you'll be participating in needs to be identified (almost definitely not the case). Anything less and students will find themselves financially and personally committed to a course of education without truly knowing what they'll be required to do to pass.
If you're interested in the topic, take a look at this neat little booklet: https://oprs.usc.edu/files/2017/04/Informed-Consent-Booklet-...
I don't see much ability to innovate here other than Mechanical Turking out the work to people you pay to participate, but if you come up with something interesting I'd love to hear it…
You'd be surprised how common that is.
I'm use to the alternative being a research paper that can easily take 20 times longer to complete than being a test subject.
I'd add in Beagles, Capuchins, Zebrafish, and Mongolian Gerbils there too.
The entire field is murky because it's very hard to scientifically measure human psychology and behavior, and the tools of the trade seem almost laughably simplistic (like the aforementioned 5 point rating questionnaires). So our body of knowledge doesn't even reliably describe WEIRD people. Rather its a crude proxy, that just might contain some elements of truth warranting further investigation.
But better crude tools than no tools, and better locally available subjects than no subjects (because most studies don't have the budget to go to Zambia). Similarly in other fields, research is done on rats pigs and monkeys with conclusions drawn to humans. Obviously not perfect, but again it's at best a starting point for later studies.
I think the real problem is the over-zealous interpretation of study results as "truth".
If you're trying to measure a target 1cm across from 1 mile away with a ruler and a squint anything you say is not only likely wrong, but woefully deceptive.
My view is that this shouldn't even be attempted because it just generates superstitious theories. Psychology is the skinner box pidegon.
There's a fallacy here that's hard to pin down clearly but roughly: to measure inaccurately isnt to measure approximately. Its to measure totally in error.
The errors in social psychology are not just "second decimal place", they're angels pushing stars.
Unfortunately, much like with more physically oriented doctors, opinions and practices will vary widely (including going against established research for whatever reason)-- and those seem to be particularly diverse in this field. The replication crisis clearly shows a strong need for reform at the research level as well.
In my opinion, the way we approach these issues should be 99% focused on what we know facilitates better outcomes (short and long term) for actual patients in a cost effective way. Theorizing about fun new methods is exciting, but what we need most these days is more disciplined and well trained pragmatists in the field, who can put their world views aside for a while to provide basic care for the (emotional equivalent of) bleeding gunshot wounds in front of them. How we get there is a big question, but one part of it is always going to be recruiting and properly equipping a certain percentage of incoming students to do the work.
I take talk therapy to be a kind of practical skill in regulating the emotions of another human being through narrative & interpersonal contact. That requires a lot of practical training, and is in fact, mostly practical training + high empathy.
Research social psychology does not inform this at all, being a totally different area.
"Psychoanalytical research" is even more BS than social psychology, but that's rather pre-advertised. And somewhat defensible as providing a "training ground" for learning the practice of talk-therapy rather than any kind of genuine explanatory framework.
Both require something to be true, but have no content on what that is.
Surely, the solution isn't to throw up our hands in the West and be like "we can't find any help for the mentally ill because some subject might not apply to our discoveries from around here!"--but of course, if they are extrapolating those results to other cultures, that's worrisome and inaccurate in many studies.
Human behavior has some universals, but a lot we figure are universal aren't.
There will always be a conflict between what we can know with confidence and the decisions that we need to make to handle situations that arise. But it has been shown that having someone in the position of authority as a 'scientist' giving their approval to something based upon insufficient evidence or reason tends to often lead to the most severe and large scale suffering. So I'd have to recommend against giving any credence to any crude tool.
"Of the authors who selected and defined the DSM-IV psychiatric disorders, roughly half have had financial relationships with the pharmaceutical industry at one time"
The point of this is that the average study in psychology is much more likely to be wrong than right. And so by indulging psychology you are not giving yourself a crude tool for understanding, you are actively misinforming yourself! Imagine I wrote a newspaper where 64% of the articles were fake or misleading. If you'd like to be as well informed as possible, you'd be better off never reading that paper, even if there are some true things in it.
Science is not a 0 or positive game. Bad science can and does send societies and progress backwards.
> A report by the Open Science Collaboration in August 2015 that was coordinated by Brian Nosek estimated the reproducibility of 100 studies in psychological science from three high-ranking psychology journals. Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies.
Journals are filters to cherry-pick 'study space'. By the way they're constituted, they publish new studies that have overstated significance.
Oh god, having filled out a bunch of these for diagnosis and such I hate these with a passion. I always wondered how well these actually work.
I've seen grammatical nonsense like, "Do you often do X? -- always, often, sometimes, rarely, never". What, I often rarely do X? And what does often mean, anyway? Like once a week? Every day?
Then, there are the abstract or vague questions that you then have to interpret what concrete situation it could apply to. Hard to think up an example off the top of my head, but how people reply to these surely depends on what exactly they think it might mean.
Then you start losing patience after about 3 minutes of this shit, not to mention 15 or 30 minutes, and just go through them barely reading the questions, but for the first couple of questions you were pondering whether you "agree" or "somewhat agree" for ages.
I grant that with a questions like "Do you often do X?", examples are necessary to specify what "often" means.
From the article:
> Some people may refuse to answer. Others prefer to answer simply yes or no. Sometimes they respond with no difficulty.
That just sounds like some people boycott the Likert questions, but we don't know why.
In a normal situation I would probably refuse and ask them to clarify, or challenge their assumptions or something. In this situation I just try to get over my aversion to answer and try to interpret it as best as I can. Sometimes I got no clue what it could mean and choose neutral or whatever. It's mentally exhausting.
This doesn't invalidate the methodology. Just saying I don't like it. If people refuse to answer or answer in a way that doesn't fit the schema and they throw it out, that's obviously a problem though.
Edit: To maybe answer your question, I skimmed through the actual article and they mention things like old people not culturally accepting a young person administering questions, children thinking they should not speak in the presence of elders. But also other weird things as mentioned that seem to me as stemming from confusion. I mean, you get better at answering with practice, so the whole concept might be totally baffling to some people which haven't gone through a Western school, where you're also quite often expected to answer unclear and confusing questions on tests.
More edit: The PhD students and professors designing these questions and the college students used as test subjects are basically the most overschooled people on the planet. They are the ones that, in school, excelled at answering abstract, underspecified, questions with multiple choice answers. Many have seen these before. They might even know how they are made and how the results get processed. Of course they're going to have less trouble answering.
Then there are the 'Have you quit beating your wife?' questions, posed to make it a dichotomy but I'm somewhere else entirely.
My conclusion is that the usability and accessibility of empiric testing could use some work. Even in regards to WEIRD people :D
I'm not even sure how you test something like that for reliability.
Not saying it's right, but not saying it's a singular question "do you believe X agree? slightly agree? etc." it's a bit more deep and nuanced than that. And the statistics tend to back it up.
I think improving these things is difficult, because of you change the questions, you also loose all the accumulated data, and maybe now you might have a better questionnaire, but you wouldn't know it, and you wouldn't know what the results mean. I mean, until you administer it to just as many people as the old one. It's a lot of work.
The amount of armchair criticism of this sort of thing always surprises me.
There's literature on the the number of response options and it basically says:
People respond quicker with binary options.
You lose information though. People get pissed there's no nuance available (no "maybe" for example). They refuse to answer because they can't say anything other than thumbs up or thumbs down.
As you increase the number of options, your ability to predict increases. Predict the same thing later, predict other things, etc. It stabilizes though.
It does take longer for people to respond, but then they complain about what "slightly agree" means.
Regardless, though, you can ask people to do it, and if they just do it, you can predict from it.
Even when you let people omit responses and refuse to respond, you can model that refusal to see what it predicts and then use their refusal anyway.
You can model how people use the response options in different ways, and in general it doesn't matter too much.
You can even eliminate rating scales altogether and use entirely different systems (forced-choice between different statements), but those don't actually work much better either.
Yes, you can always respond in a cheeky, subversive,or manipulative way, but then that's an entirely different issue altogether. You can always do that, and all you've shown is that you're smarter than a rating scale?
As for the original article... these criticisms have been made for decades. Decades. These are my general impressions:
1. The problem of western focus in behavioral sciences is a problem of western focus in the sciences period. Many behavioral scientists would love to do research across multiple sites but cannot afford to do so. And this doesn't stop all sorts of other biomedical research from being done on very narrowly defined groups of people (or animals).
2. Effects observed in undergrads generalize a lot more than the author is letting on, and effects observed in western populations generalize even more. I'm not saying it's not important to study things cross-culturally, only that the idea that people are fundamentally different in different settings is itself flawed. Many things have been examined across different cultural settings, and the differences are not all that dramatically different. In fact, in one recent replicability study, the effect of culture/sociogeographic population was one thing that didn't seem to matter that much. Some studies replicated and others didn't, but the sociogeographic setting didn't seem to matter very much.
I agree that being more sensitive to human variation is critical, but like a lot of things with behavior, there's a lot of grey areas, which people don't like to hear.
There are whole fields within psychology and the behavioral sciences devoted to these issues. People have put a ton of effort into studying them, considering all the issues being raised here as well as many others to numerous to count, and it's like all that work gets brushed aside like snow in the wind because of random blog posts and anecdotal experience.
And what about the 'Have you quit beating your wife' part? Where I don't lie on any part of the spectrum they've drawn. Ok, that's the same with thumbs-up-or-down, but with the spectrum its in your face that you have no answer that's meaningful. I refuse to answer those kind far more often that thumbs-up.
Finally, mega-corporations wanting actionable results are not 'brushing aside with anecdotal experience'. That's the point I made. They're spending billions and want results, and more often use thumbs-up-or-down. That trial has five orders of magnitude more data than all the graduate students in history added together.
But yes, all these studies whose the authors authoritatively make claims but these are are based on people answering questions regarding themselves (how often do you cheat on your partner?) - I never believe these numbers.
I wonder how much shrinkage (wasted questionnaires) they have to account for? Pun semi-intended.
Edit: to clarify, there are many other questions that will seemingly be somewhat related or unrelated that will infer later on in the test if you were lying. People who aren't familiar with the tests won't notice it in an honest testing.
Especially since a study with 300 participants is worth immensely more than 10 studies with 30 participants.
The worst is it’s contextual loyalties against critiquing the relationsips people have with one another. Power struggles, to be particular.
Effectively, clinical psychology serves apology for societal and economic ills by blaming the effects on the individuals themselves.
The process is strikingly transparent from an objective standpoint, but ironically it’s precisely an objective standpoint that is undermined by the institution. These days, you can visit one and tell them just about anything, and they will diagnose your disposition as illness.
I don’t blame psychologists themselves, and who is to say they don’t help people? Many surely do, but the institution is an absolute drag on our economy and society; something like a collective excuse for avoiding self-critique.
Sociology theoretically steps in here, but it is merely a study, and some branches of it would result in a totalitarian nightmare.
What’s needed is humanist sociology and humanist psychology. As we learned in kindergarten class, there is nothing wrong with us. Each of us are unique and beautiful in our own ways. The questions are about how to live with one another.
Pyschology as a study is another thing, but as a medical malpractice it has proven antithetical to systems design from the start.
In the meantime, I would think the field of clinical psychology is a safer bet.
It's common sense, and
there is no objective proof to the contrary.
"Psychiatric diagnosis still relies exclusively on fallible subjective judgments rather than objective biological tests"
Psychiatry/psychology also admits that social environment has substantial effects on mental illness.
But asking for a "scientific" study of that is silly, because it is impossible to have an indepedent variable or control group -- because it is impossible to separate an individual from societal influences.
I'm merely claiming they're not being scientific.
There is plenty of evidence that disfunctional social environments often lead to disfunctional individuals.
It's not like this is some groundbreaking controversial revelation that flips the bird to science. It's a fact that isn't incorporated into a model of mental illness treatment that pretends that "mental illness" is primarily a result of a disordered badly behaving brain. This model is used because its easier to change a person than the society around them. It's easier to empirically measure change when you're measuring the effects of something on an individual rather than on the entire society.
I would account for the decline in mental health metrics in the west, there's more self reported mental illness, more suicide, more social withdrawal, largely on the failure to address big picture concerns. Clinical Psychology isn't useless to solve the mental health problems of society. It's just inadequete. It's like expecting tier 1 support to solve a problem that's rooted in an entire network being fucked up. What I see is a society that has problems that are correlated with poor mental health which aren't getting addressed and people are acting as if the solution is leaning more on solutions like talk therapy and pharmacuticals which are failing to address a society-wide decline in mental health.
While speaking against e.g. mainstream cultural tenets, or how we run society, or how we parent, etc., can lose you grants quickly, and those are all things psychology and sociology can get at.
I don't have any issue with the general message. This surely is a legitimate problem. But the intro reads a bit like "psychology is such a great endeavour, if there wasn't this little issue."
Everyone following science news should be aware by now that psychology suffers from a whole range of systemic methodological problems, notably publication bias, widespread p-hacking and failed replications.
Think children growing in warzone or poor and violent environment. Their behaviour as adults is often sexually more promiscuous, aggressive and their impulse control seems to be less than 'the baseline'. They show trust issues.
How much of that is just damage and disorder as psychology seems to assume, and how much is adaption to survive and procreate in an environment where lifespans are short and life is uncertain. Maybe childhood stress and stress hormones trigger survival strategies that work well in hard environment. They are maladaptive only in the culture and safety of the developed world.
It would mean we are effectively blind(er) to unfamiliar shapes, even though they are extremely simple, like a triangle or square are.
It could have something to do with their written language, for example, and shapes similar to triangles and squares in it appearing in a certain pattern that isn't the basic repeating XXOXXOXXO...
Or it could be music and rhythm is a big part of their culture, and XXOXX then follows with XXXO in a large number of their songs.
Those would have been things recognizable to me. Things I liked a lot.
Also, they would have put my mind somewhat at ease, I think.
I wonder how much of the inability to answer some questions is actually down to feeling slightly intimidated by the onslaught of questions, their sterile aura, not knowing the person presenting them, etc...
How much does the subconscious feeling of intimidation influence the ability to think?
Of course, it's always possible any individual study contained unknown and unaccounted biases. Funding in science is still relatively small, the number of people participating few and science itself is hard.
So just by simple fractions we know something is off. A more careful study by subject and region could be required.
They should be considered temporary interpretations of statistical data or in short meta statistics cause that's really what they are.
Much damage is being done by treating these fields as science and the article is only mentioning a few of those problems.
This is what a lot of experimental sciences are, even physical ones, when the systems being studied are complex.
There are very few areas of scientific study anymore that offer convenient, deterministic results. That fruit was picked a long time ago.
Even at the cutting end of physics, researchers have to infer from statistical results.
The difference is only that some of these fields have more reproducible results than others, often because they are studying less complex phenomena, whose causal factors and mechanisms can be more directly observed.
Psychology is at one end of that spectrum, because it is studying the output of the mind, a biological information system whose mechanisms are among the most complex and obscure that people have ever studied.