It’s been a while since I was studying this topic but if I remember correctly, there is fundamental problem of a lack of justification for operational definition in the whole field of psychometrics. This applies to most (all) personality tests including the big 5.
Because the whole theory is based around cluster analysis it is pretty important that you can justify the data that is going into the analysis if you want it to be reveling any truth outside of the model, otherwise you end up with what is called “junk-in-junk-out”. As far as I know, this justification is still lacking 40 years after this theory surfaced.
I think that the big 5—and personality psychology in general—might not have the same glaring issues as positive psychology. But solid science it is not. Fraud might be overreaching, but I would definitely categorize it as pseudoscience.
That's pretty subjective and could be leveled against almost anything behavioral or psychological.
The thing about the Big Five is that it has surfaced in all sorts of contexts. You can maybe say maybe the 5 per se is not well justified (as opposed to 4 or 6 or 7, for example), but if you take enough ratings of a person, some variant of those 5 will probably work as reasonable summaries of the ratings, and they will account for a substantial chunk of their predictive variance. The thing too, is that if you take other types of variables, like clinical symptom ratings, or diagnoses, you start to see roughly similar types of attributes become prominent.
The Big Five is a descriptive model of how people perceive others. There's a lot of evidence for certain mechanistic processes being heavily involved in some (e.g., positive emotion in extraversion, negative emotion in neuroticism, behavioral control with conscientiousness, etc.) but I'm not sure the original idea behind the Big Five was mechanistic -- it was a hypothesis about major dimensions that could summarize social perceptual data. It's like classification in biology pre-DNA era. People have some ideas of how things go together, and find it useful for organizing descriptions and measuremnts.
It's like if you did unsupervised DL modeling of all the videos involving humans you can find on the web, and found that their classification could be accounted for by 5 major vectors, almost all the time, regardless of sampling. Wouldn't you want to know that?
Many other measures are very mechanistically well-justified but lack ecological validity, in the sense that they are very narrow predictively and not well outside of laboratory contexts. That's fine, there's a tension there between predictive bandwidth and depth, but if you want any kind of rating of a human being's behavior and experience, you enter at your own risk if you think you'll measure something that's radically different from the Big Five (or something subsumed by cognitive measures). Can you do it? Sure, but a lot of the time no (see: grit).
What constitutes as an adequate justification for use of an operational definition within a model is subjective indeed. However there is usually a point in the life of a theory where the gathered evidence are sufficient enough that a scientific consensus starts to form that the operational definition is justified. I’m not aware that that has happened in the 40-odd year history of the Big 5 personality theory.
The 5 personality traits may be overarching within the field of psychometrics and they may indeed be useful to describe behavior, however you still need to justify that said behavior is not easier described using different models, and this is where personality psychologists usually fails in justifying their operands.
Works criticizing the model range from using totally different constructs (such as priming, positive reinforcement, universal grammar, brain dopamine level socio-economic status etc.)—which don’t rely on psychometrics at all—to claiming that the behavior psychometricians are predicting are actually not that useful (e.g. predicting ‘high confidence’ is not that useful if ‘high confidence’ does not result in a significant behavior which isn’t better predicted without made up operands).
If you were an early astronomer and you constructed the notion of ‘epicycles’ to simplify your model of planetary motion. You may use these ‘epicycles’ to justify your prediction, however you may not use a successful prediction to justify the existence of epicycles. Your epicycles may be useful until someone comes along and deems them unnecessary since planetary motion is better described by using elliptical orbits.
Of course this could go the other way as is the case with particle physics and the atom. However given the amount of research, success of rival theories, the failure of psychometricians from making useful predictions outside of their narrow field that isn’t better explained with alternative theories, I have high doubts that the Big-5 personality traits (and any theory of personality using psychometrics for that matter) is anything but pseudoscience.
> If you were an early astronomer and you constructed the notion of ‘epicycles’ to simplify your model of planetary motion. You may use these ‘epicycles’ to justify your prediction, however you may not use a successful prediction to justify the existence of epicycles. Your epicycles may be useful until someone comes along and deems them unnecessary since planetary motion is better described by using elliptical orbits.
I couldn't comprehend the discussion until I read this metaphor. Thanks for the detailed explanation.
I appreciate your response, but think I'd have to read more to be nearly as convinced as you. I'm familiar with the critiques you mention, but your conclusion as to the model's merits go beyond what I've seen other critiques make. As a model grounded in theory it is lacking, but as an explanatory construct to predict patterns of behavior and outcomes in specific settings (e.g., knowledge work) it persists for a reason. I've peer reviewed papers criticizing it, but none "debunking" it.
Personally my biggest gripe personally is that it is represented as a model of total personality, but that's definitely not true. It's just just representing a larger "personality space" than most other constructs. At least the outcome isn't placing one into a discrete category.
When I was doing psychology over a decade ago personality psychology was actually my biggest gripe. The way that I saw it, was that it wasn’t explaining anything which didn’t have better explanation using a different theory.
Now a decade later—being a little more class conscious—I can actually see how this is problematic. When a better explanation can come from sociology and has to do with class and economic status, measuring people based on a theory that lacks justification to justify one hire over another is problematic. When you ascribe it was because of an operationally defined concept (read: made up; based on data analysis) and name it personality that seems like a lame excuse to hire from your ingroup and excluding your outgroup.
These tests are robust, I know that, however robustness alone is not enough to justify a theory. These tools might be useful (or they may be dangerous) but while there is no justification for the operationally defined terms it cannot be used to justify terms outside of the model. That is pseudoscience. You can only to explain things inside of the model, and as such it is pretty limited as a theory.
I agree that the label "personality" unreasonably implies some sort of real (possibly for many, "genetic") truth.
> When a better explanation can come from sociology
Explanation for what? Behavior generally? In that case, the Big 5 is less of the predictor and more the criterion. The Big 5 is not a theory, it's a taxonomy that emerged from trait theory. It describes, it clusters, it predicts, but in and of itself it doesn't explain.
> ... seems like a lame excuse to hire from your ingroup and excluding your outgroup.
When it comes to hiring in the United States this is literally the opposite of the biggest use-case for personality testing for the past 50 years. But that's the only domain I can speak confidently on, and that doesn't include class or SES as a subgroup.
> there is no justification for the operationally defined terms
This is subjective, and I do not agree. If your critique of The Big 5 is actually a critique of trait theory more generally, I'm there for that. But a taxonomy for observable behaviors that shows reliability as well as content, convergent, and predictive validity across many populations and contexts seems justified to me barring a superior option.
The Big 5 is not and never would be a grand theory of human behavior, but it does describe actual behavior in a way that is interpretable and connected to the world we actually inhabit.
I don’t know how people interpret results from personality tests which gives them a criterion for what constitutes a good hire. However I have a strong feeling there is no sound science behind whichever criteria recruiters are using. And that risks placing arbitrarily high weights on whichever traits correlate with your in-group. This, however, is a falsifiable claim, and if anyone has ever done research which shows that personality tests actually reduces bias (as opposed to enhances it) then I’m open to be proven wrong. I do also question the efficacy of using personality tests as a tool to scope for good hires. How do companies actually measure that? There are so many biases that comes to mind that could make a company overvalue the efficacy of these tests.
My critique applies for all theories of personality. I don’t see personality traits as a useful categorization to predict behavior. By far the most research I see in personality psychology is about correlation with other terms inside a very narrow scope (this also applies other sub-fields of psychometrics; including positive psychology). The behavior that personality tests predicts does not further our understanding of the human mind. A theory of behavior that fails to do that is a poor theory at best.
But I want to go further, I don’t claim that personality psychology is just poor science but pseudo-science.
>> there is no justification for the operationally defined terms
The personality traits in the big-5 are operationally defined. That is they are defined in terms of what the tools are designed to detect. This is useful inside your models (as evidence by the success of big-5) but this does not tell us anything outside of our model. Now if you go to the real world and find evidence that these terms exist outside of your model, then I would change my mind. That would be a pretty good grounds for a theory which describes something that your model predicts accurately. If you don’t then you have at most a useful construct that you can use in other theories (think atom before they proved it’s existence). IF you can’t even use your constructs outside of your scope, then there is not much value in it outside of a narrow scope and you are most likely doing pseudo-science.
My critique actually extends to all personality traits. Personality traits (if they exist) can at best describe a proportion of the variability within a narrow scope. Outside of that narrow scope it is actually just better to describe a person as calm and courteous as opposed to speculate where they stand in the agreeableness axis in the Big-5 personality trait. And this is what I mean when I say “lack of justification”. ‘Agreeableness’ has been operationally defined within a certain model. When you use it elsewhere you need to justify doing so. And there should probably be a scientific consensus about whether the justification is good enough. If not you are most likely doing pseudo-science
Note that "operationalization" implies a fairly specific set of epistimolgical and ontological approaches which do not necessarily require that what is being measured has a one-to-one correspondence to a 'real' entity.
Indeed. You can operationally define anything you want within your model. If done carefully, a good operational definition may simplify your model quite a bit. (A bad operational definition, on the other hand, will almost certainly make your model overly complex and can be quite detrimental).
When you use your model to infer about a real world phenomena you have to be careful how you treat your operational definition. If you use it to make prediction, you cannot make a claim that what your operand caused it, not until you go into the real world and find it. If your model is successful you may use your operand to describe your prediction, but you have to justify why your operand is necessary, a better model may exist which doesn’t use an operand at all.
A successful model is neither a sufficient nor necessary condition for proving an operand exists.
Because the whole theory is based around cluster analysis it is pretty important that you can justify the data that is going into the analysis if you want it to be reveling any truth outside of the model, otherwise you end up with what is called “junk-in-junk-out”. As far as I know, this justification is still lacking 40 years after this theory surfaced.
I think that the big 5—and personality psychology in general—might not have the same glaring issues as positive psychology. But solid science it is not. Fraud might be overreaching, but I would definitely categorize it as pseudoscience.