
Let me automate that for you: removing pain points from data pipeline setup - fanf2
http://tech.gc.com/let-me-automate-that-for-you/
======
dblock
I like this. A nice, practical solution to a real problem!

Disclaimer, I work for AWS. At some point there's just too many of these data
streams, so a warehouse no longer does it and we turn to data lakes. The
advanced version of what the OP is trying to achieve, without any actual
manual work, could be an AI, and be called Lake Formation,
[https://aws.amazon.com/lake-formation](https://aws.amazon.com/lake-
formation).

------
nabdab
Really great story. I feel like the same low key problem at my current
workplace would instead have spawned a “let’s replace everything with a data
lake”-project, spawning several years with an army of consultants and at least
one new major external platform. All to solve these small pain-points that you
can weed out if you have a passionate and talented team who are close to the
problems and have freedom to build the solutions.

------
_frkl
Nice writeup. I've used [https://www.singer.io/](https://www.singer.io/) to
good effect in the past, seems like this is a similar, maybe more
manual/custom approach.

