Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Are there theoretical arguments to why random sampling is bad and one should instead use the entire dataset ?


Random sampling is the foundation of statistical inference.

It works really well for estimating population statistics from sampling.

It works less well for predicting individual user behaviour (this tends to be where you need almost all the data rather than a sample).

Like, as your sample size decreases, your errors of estimation increase (1/sqrt(n)) so random sampling is normally the best way to approximate data (e.g. for real-time metrics).

You may wish to consider different sampling strategies, however. For some things, a persistent user (or other dimension of interest) makes lots of sense. For tracking service health, you probably want to sample on requests.

You may also wish to over-sample particular groups, if they are important to you (like maybe an F2P game would sample for most users, but store a larger proportion for paying users).

tl;dr random sampling is awesome, and should be used broadly.


Not sure if that answers your question, but:

If you are looking for seldom-occurring patterns, your random sample almost certainly won't contain it.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: