On: The new local data stack: Integrating Dagster, dbt and DuckDB

[OJFord]

Tools like Dagster, dbt, and AWS Glue always strike me as built for BI/in-house analytics over user & app-generated data.

What do people use for ingesting data from end-users, their transactional CRUD to it too, and then OLAP queries triggered (not written) by and for those same end users?

[benjaminwootton → OJFord]

There are a bunch of collector tools for capturing application side events and sending them into your server.

The common stack is then to ingest them from Kafka into an OLAP datastore like ClickHouse or Pinot.

Then I think you are referring to what is sometimes called “user facing analytics” against that datastore.

We like to use a tool called Cube.dev as middleware between the app and OLAP database which gives APIs and a semantic layer.

[geoHeil → benjaminwootton]

Not to forget https://www.starrocks.io/ or https://risingwave.com/ - see also a recent blog post of a friend: https://juhache.substack.com/p/moving-from-bi-to-data-apps-p...

[OJFord → benjaminwootton]

'user-facing analytics' and (via Cube.dev) 'embedded analytics' are helpful terms, thank you!

[totalhack → benjaminwootton]

+1 for using a semantic layer. I built (and open-sourced) one for use in my own business but Cube looks well done.

https://github.com/totalhack/zillion

[gms → OJFord]

For the ingesting bit, we (Polytomic) have an API you can use: https://www.polytomic.com/connect.

[lysecret → OJFord]

Bunch of python code.