

Research HN: Is there a data science DevOps? - joeclark77

I&#x27;m looking for people who are working in data science, big data analytics, business intelligence, and related areas, who would be willing to be interviewed (confidentially if you like) about how this type of work is best done in teams.  I&#x27;m particularly interested in the infrastructure or the DevOps, so to speak.  We know a lot about the operations management that supports regular software development teams, but I haven&#x27;t found any good guidelines for which (if any) tools and processes play a similar role in supporting analytics work.  Does anyone here have any thoughts or experience with this problem?
======
quantisan
I was a data engineer for a couple years at one of the largest websites in
Europe. My job was to develop applications to handle our data pipline and
deploying the infrastructure on Amazon EC2, Redshift, EMR, etc.

What is it that you need and what are the problems you're facing?

~~~
joeclark77
I'm coming at this as an academic, and the problem I'm hearing from companies
about the whole idea of a "data scientist" is that it describes a superstar --
someone who knows math, statistics, programming, the business domain, etc, and
is self-sufficient. The glowing articles about data scientists focus on the
talents and skills they have, and don't tell us much about the operational
"work" they do. I can't tell from the literature, for example, what kind of
workflow I'd see on a kanban or scrum taskboard for a data science team.

Closer to home, I am also very interested in how a university can train
students to do this kind of work... so I want to understand the operational
processes and not just a list of technologies the students should learn.

I'm currently teaching a course on software development which touches on
workflows and DevOps, and I'd like to do the same for our new analytics major.

~~~
quantisan
You mentioned that you know about general software development devops already,
why and what do you think is different for data science workflow?

------
w_t_payne
Yup. Loads. reach out at wtpayne-at-gmail-dot-com if you want to talk.

