Hacker News new | past | comments | ask | show | jobs | submit login

The problem is DS is really 2-3 different disciplines under one nebulous title. What you're describing is folks who are prototyping and productionizing models. That's definitely in short supply, but random STEM PhDs are in no way competitive for those roles unless they're coming from CS programs + have work experience in production engineering.

But that's by no means all of the DS field. There are lots of DS jobs where you're collecting and interpreting and communicating about complex data sets. An engineering mindset is occasionally helpful, but a bias towards building versus towards analyzing and writing can just as often be counter-productive. Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution.

The bootcamps have contributed to the problem by focusing so much on building things. The idea that you can go from an econ undergrad to being a self-sufficient member of a production ML team in 6-12 weeks is nuts. What's less nuts (and what I wish programs like Insight focused on) is taking people from having data skills in one domain and with one set of tools (e.g. logitudinal medical record data, stored in CSVs and handled in Stata) to another set of tools (billions of rows of event-based product data stored in a data warehouse, processed in R or Python). But instead the bootcamps behave like the missing skillset is the ability to make a predictive random forest model on some arbitrary data set and build an AWS web app around it. THAT job market definitely doesn't exist and is completely over-saturated.

But people who are smart communicators about data, can manipulate and make sense of massive data sets, can ask incisive questions about their data, and can use data to convince people of a complex argument are always going to have job opportunities, even if they're not production grade engineers. If that sounds like you, I'm hiring - hit me up on Twitter: @drewwww.

"Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution."

Agreed wholeheartedly. Reminds me of another quote from http://www.john-foreman.com/blog/surviving-data-science-at-t... :

""" You know what can keep up with a rapidly changing business?

Solid summary analysis of data. Especially when conducted by an analyst who's paying attention, can identify what's happening in the business, and can communicate their analysis in that chaotic context.

Boring, I know. But if you're a nomad living out of a yurt, you dig a hole, not a sewer system. """

I was part of the Data Science team in my previous company. We mainly build models for production, but we also were responsible for generating both daily and ad hoc reports. We tried to hire someone to take over the reporting part, but we found out even that requires engineering skills. This role ended up to even more difficult to hire for because it's hard to find someone who has the engineering skills but wants to work only on reporting. Maybe if we had a dedicated data engineering team the story would be different.

That is me. A decent developer who gets data and enjoys reporting, particularly the variety of hard problems that crop up.

I am happily serving a niche market with my own company and I suspect part of the difficulty in finding the skillset is that we can just start our own thing when we find a domain that we like.

Yeah, in my org we have a dedicated data engineering team that owns the pipeline and production data systems plus an analytics team that owns reporting and making data available to non-technical data consumers. That leaves complex research work for data scientists who are (in theory, anyway) building on top of stable data infrastructure and good data sources and free from ad hoc reporting work.

In France, and other Francophone countries, Statistics and Information Analysis is an engineering discipline.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact