This is many orders of magnitudes better confidence than any physics experiment, it feels unlikely a biological result can even be this strong, so it makes it sound like a statistical error.
Bioinformatician here. These kinds of p-values are common in these kinds of experiments (GWAS, or association studies), and happens almost automatically once you get enough statistical power.
The big problem is that once you have so much statistical power, you get very small p-values from small effects, and then the often-overlooked assumptions behind frequentist statistics begins to matter. Are your samples _really_ idenpendent and identically distributed values? No. Are they really normal distributed? No.
Also, things like gut microbiome and depression are linked through what some people call the _crud factor_, which are weak correlations between nearly all social aspects. For example, probably depressed people eat differently from non-depressed people, causing changes in their gut microbiome. Probably, there are variations in human population's depression rates and obesity rates (correlated with gut microbiome) that somewhat correlate.
When you have enough statistical power, you see the crud factor everywhere.
I really appreciate your explanation regarding the crud factor. It adds a lot of intuition.
Out of curiosity, what is your perspective on gnotobiotic systems? I distinctly recall an example of a gnotobiotic mouse that, upon being provided a microbial sample from an obese mouse, started to have substantial increases in body fat despite a simultaneous reduction in feed. Would that type of experimental approach still run into the statistical difficulties you mentioned?
Well no. It's not uncommon for p-values to be even lower than that. We are talking about a specific SNP (allele) having a specific mutation being predictive of a phenotype / outcome.
So, a specific SNP mutation being predictive of a gene expression / protein is basically a p-value of 0.
Can't speak for physics experiments, but this is almost certainly not a statistical error
This is many orders of magnitudes better confidence than any physics experiment, it feels unlikely a biological result can even be this strong, so it makes it sound like a statistical error.