
Lack of group-to-individual generalizability is a threat to human subj. research - _1tan
https://www.pnas.org/content/115/27/E6106
======
_1tan
Abstract:

Only for ergodic processes will inferences based on group-level data
generalize to individual experience or behavior. Because human social and
psychological processes typically have an individually variable and time-
varying nature, they are unlikely to be ergodic. In this paper, six studies
with a repeated-measure design were used for symmetric comparisons of
interindividual and intraindividual variation. Our results delineate the
potential scope and impact of nonergodic data in human subjects research.
Analyses across six samples (with 87–94 participants and an equal number of
assessments per participant) showed some degree of agreement in central
tendency estimates (mean) between groups and individuals across constructs and
data collection paradigms. However, the variance around the expected value was
two to four times larger within individuals than within groups. This suggests
that literatures in social and medical sciences may overestimate the accuracy
of aggregated statistical estimates. This observation could have serious
consequences for how we understand the consistency between group and
individual correlations, and the generalizability of conclusions between
domains. Researchers should explicitly test for equivalence of processes at
the individual and group level across the social and medical sciences.

Why I posted this to HN:

I think this is another important finding underlining how much of our current
research is broken. Discussions about similar issues like e.g. P-hacking on HN
were highly interesting. I believe this another aspect of the same core issue.

Note:

I had to abbreviate "subjects" to "subj." due to the HN title limits.

