On: A modern data stack for startups (2022)

[jimberlage]

Out of curiosity, would running DBT that outputs to a reporting schema on a Postgres read replica work? Or as a startup do you already have too much data for that?

[lawrjone → jimberlage]

It would work for most of our business reporting, but BigQuery is so good at what it does and cheap that it doesn't make much sense to avoid it.

We also have some parts of our dataset that are complicated to compute.

An example is that we estimate the hours that people have spent working on incidents, with a method that we describe here: https://incident.io/guide/insights/workload

That requires us to do some expensive cross-joining of every action ever taken in an incident from messages sent to the channel to GitHub PRs being merged. We could make this incremental and optimise it for performance but using BigQuery by default means we don't need to worry yet and can leave the optimisations for when we're bigger and the engineering resource we'd dedicate wouldn't detract as much from customer-focused work.