
Ask HN: Data Scientists / Statisticians of HN, how do you verify your numbers? - lost_sqler
Hi everyone. I&#x27;m in need of guidance as to how to verify the correctness of my work.<p>I&#x27;m a trained software engineer, and spent most of my career programming applications, of various kinds: mobile apps, websites, backends...
About a year ago, I got a new role that requires me to sift through quite a lot of data and extract numbers from it.
Basically, we have a big SQL database full of events, and I&#x27;m responsible for building reports based on those events.<p>That&#x27;s a lot of fun! The database has grown to a size where it actually becomes a challenge to run those big aggregations,
and I must tweak my queries carefully. 100% recommend to anyone interested in... well going through a lot of stuff.<p>However there&#x27;s one aspect of it that&#x27;s a lot less fun: I find it hard to get a &quot;yes, those numbers are correct.&quot;.<p>At first, I assumed I could simply write unit tests, reproduce the data that we should have in the production database,
and verify my numbers on smaller datasets.<p>The reality, however, is far from that. While the events respect established schemas, some were lost. Others didn&#x27;t arrive in order.
Some are duplicated, and others seem related but actually are not.
Also, due to the sheer volume of records, I can&#x27;t just start counting them manually.<p>I realized, on multiple occasions, that some queries were incorrect. Of course the resulting numbers were incorrect too.
But we didn&#x27;t notice that immediately, and some reports had to be recalled&#x2F;corrected. No harm was done, but I did lose some sleep over it.<p>So there&#x27;s that. While I very much enjoy working with that data, I wish I could find a way to validate my work,
and finally be able to click on a &quot;resolve&quot; button knowing that my results truly are correct.<p>Is there any magic trick I&#x27;m missing? Any tool that could help me? Any workflow that would be my data unicorn?
======
euix
The trick is to realize the act of verifying the accuracy of the data "is" the
science. I come from High energy physics background. People think the point is
to find sexy new particles or make ground breaking discoveries. No the real
goal is to understand the inner workings of every part of your data
acquisition pipeline (from the hardware detector to the software analysis) so
well that you eliminate all possible other explanations when you spot a
deviation. That then is your discovery.

I was a DS for a couple years in NYC area. My advise is the same as in
science, treat the act of verification as the end goal itself. If you
understand your experimental apparatus, in this case, how the DB is populated
then you understand all subsequent of the "physics". Anything you can't
understand should be carefully quantified as random and systematic errors.

Sorry I don't have any magic bullet or standard recipe for how to go about it.
Every case is different.

~~~
lost_sqler
That does help, thank you. I will look into making the verification a core
part of the process.

