
New Data Engineer, good learning resources? - willnewby
Hello!<p>I just got a new job as a &quot;platform&quot; engineer, which looks like it&#x27;s going to be a whole bunch of data engineering (i.e. getting data from point A --&gt; redshift&#x2F;etc). I come from primarily a webops&#x2F;devops&#x2F;sysadmin background, which means I&#x27;ve been feeling a little out of my depth on &quot;what&#x27;s all this data stuff and how does it fit together?&quot;. I&#x27;m curious for any good resources you may have.<p>Thanks!
======
mattbillenstein
That is a hard question to answer, I think you have to start looking at the
big component (the data warehouse) and see what fits best for you.

At least what has worked best for me is using BigQuery and building custom
tooling around that. Using Airflow, some simple python tools, .json.gz
archives, and a mix of streaming data and hourly/daily exports has worked well
and not been a huge task to keep operational. Of course, every situation is
different; ymmv.

~~~
willnewby
Gotcha. Good to know. Thanks!

