
Designing a Faster, Simpler Workflow to Build and Share Analytical Insights - yurisagalov
https://open.nytimes.com/faster-simpler-workflow-analytical-insights-ae6c7055e187?gi=c5b282e18728
======
teej
This seems so overengineered. If you're using a BI tool like Looker, you can
configure how often and how long you would like your "query" to be persisted.
They also support dynamic triggering - when value X changes, rerun query Y.
This is all 2-3 lines of LookML configuration.

Also, pulling this out of the dependency graph of the rest of your ETL system
seems silly. Analysts now have to schedule their queries "sometime roughly
when I think the data will be available" vs "immediately once the data is
available".

It sounds like Chartio isn't really keeping up with their workflow. So assume
for a second they need to ease this pain but can't justify a vendor switch
right now. What's the fix? I think you make it easier for analysts to put jobs
into Airflow. That way you aren't reinventing the wheel (building a completely
isolated ETL scheduling tool) and you're actually attempting to solve one of
th underlying problems (Airflow isn't "analyst friendly").

I'd love to hear other people's experience. I recognize I'm being a bit of a
negative nancy.

------
reacharavindh
Cane here to say just that. This feels wayyy over engineered. First of all, do
they really need 30 analysts pouring over viewer stats?! Isn't their primary
business journalism and producing good quality content? I wonder how many
analysts and data engineers they have supporting the articles and research..
Couldn't they just use data exploration tools like Looker and focus on their
primary business instead?

Now, I go back to my naive village wondering why The Times Digital
subscription costs so much...

