Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You're still picking on the actual, specific number, and ignoring (or otherwise completely missing) the part where I specifically said, "for sake of a convenient number". I chose 50% because it makes the math easy. Period. Full stop. End of its value or contribution to the discussion.

I have exactly zero idea what the actual ratios are. But the actual ratios absolutely do not matter for the purposes of illustrating how the methodological errors compound.



The most salient detail about the way methodological errors compound is that by engaging in more replications, you can beat down the rate of spurious results in the replications. It's of little importance that it is multiplicative with the flaw rate of the original, because the flaw rate of a set of independent replications will go to zero super fast in the size of the replications.

If the flaw rate is >= 50% and the number of replications is low, it presents a case where it's of little social value to do replications, depending on their cost (which is usually high).

From your original comment, the way a naive reader will view it is that replications often don't work, because you primed them with a number you just pulled out of nowhere (50%). That will matter much more than the other detail about it being multiplicative, which doesn't really matter much at all no matter what the flaw rates are, because it all hinges on the flaw rates of the replications and how many replications you do anyway.

I know that you did not mean to misrepresent it with the 50% number. I'm just harping on this because by happening to use a number like 50% it has the potential to distract and do more harm than good, regardless of anything else in your comment or even whether or not the 50% had anything to do with your main point.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: