I feel like human behavior will change as culture evolves. For example how people react to a plane hijacking drastically shifted after 9/11. In a similar way the Stanford Prison Experiment probably permanently changed a part of human culture so that it will no longer be replicable (regardless of whether the original setup and results were valid or not).
How do you conduct science when whatever important truth about human behavior you find invalidates the same findings once that knowledge enters popular culture?
The replication crisis has provoked many discussions on similar issues among psychologists, i.e. some dinosaurs question the value of direct replication studies and prefer conceptual replications (see for example this paper by Simons https://www3.nd.edu/~ghaeffel/Value.pdf).
I believe that direct replication is the only way to verify the credibility of scientific discoveries, and certainly most new generation psychological researchers agree on this.
In some ways it will but in other ways it won't. There are many patterns of human behavior that have not changed, ever, as far as we know.
> How do you conduct science when whatever important truth about human behavior you find invalidates the same findings once that knowledge enters popular culture?
I don't think this is the problem.
I think the problem is that we don't have a theory of the mind and therefore psychology is a field where only erudition is possible. I don't consider psychology, or other "social sciences," scientific. These fields generally lack predictive power and tend to be necessarily qualitative. That's not to say that they're useless but it's an important distinction.
I don't think it's clear that we'll ever have a good theory of the mind. Maybe understanding your own mind is like jumping over your own shadow. How can you understand the thing that allows you to understand?
Your question focuses on psychology (and I agree it's important here), but this probably happens to greater and lesser degrees when studying any complex adaptive system.
Say you go study the carbon-uptake potential of a few species of tree, model how planting more trees could impact the climate, and click the publish button.
Your research may already be out of date. Your ability to completely control variables in a complex system has limits. Most of the environmental measurements that went into your model are moving targets. Even if your model is predictive at publication, there's a good chance there are unmodeled dependent variables lurking.
What if your research captures the zeitgeist? What if you inspire the planting of a hundred-billion trees? Do your measurements consider lone trees? Does the density and surface area forested (and eventually the weather-shifts a large forest triggers) impact how it grows and functions as a carbon sink? What if your research inspires someone to plant a massive monoculture that ultimately incubates and spreads a parasite that ultimately destroys the species?
There's a fundamental shortfall in the quality of our thinking relative to the task (and the ultimate answer may just be that we're too time-bound to think this way). It's in the study designs, papers, abstracts, and unavoidable in the popular press.
1) that patients biases aren't reflected in their research
2) that their own biases aren't reflected in their research
And of course ... there's the https://en.wikipedia.org/wiki/Rosenhan_experiment
> Not at all. In terms of my intuitions being better than they were — no. And furthermore, I have to confess, I’m also very overconfident. Even that I haven’t learned. It’s hard to get rid of those things.
> I’ve been studying that stuff for over 50 years and I don’t think that my intuitions have really significantly improved.
I do expect that some experiments conducted in psychology are non-repeatable, especially if they become famous. The question isn't whether it was science, but whether the target phenomena is important enough, and stable enough, to warrant our interest.
In anthropology, Clifford Geertz famously argued that if there are laws of culture, they must be entirely uninteresting ones. His approach, instead, was rich description in place of theorization. His arguments are compelling, however, he too had this notion of science as universal law. I would argue that a model-based notion is much more productive, and much more accommodating to the problem domain. Let me go one step further: what one should focus on instead, is process by which phenomena observed are instantiated. Similar processes can result in quite different outcomes, and similar outcomes can be the result of very different processes.
I work in the field of decision neuroscience. We work, largely, by linking human behavior and neuroimaging to fitted parameters in a family of reinforcement learning models (see: Reinforcement Learning: An Introduction, Sutton and Barto). Decision making processes are largely invariant, even when knowledge, and domain-representations, might change with changes in popular culture: that is, changes in behavior can be accounted for by changing the priors, or input information, into the model.