Hacker News new | comments | show | ask | jobs | submit login

I've literally seen the opposite interviewing as distributed computing person, most people ask for a distributed engineer and then want someone to stitch together Hadoop and Spark or how to effectively leverage HDFS. When you start talking about concurrency issues, Lamport clocks, consensus algorithms, etc. their eyes glaze over and they ask how to efficiently rotate an array. Clearly something is broken but I'm not sure it is how we use language.

This a thousand times. For many firms, the roles of "data scientist" "data engineer" "distributed system engineer" and "platform engineer" are all fully synonymous, and all of them really mean "Hadoop* babysitter with a dash of full-stack whenever we arbitrarily feel like asking you to do other stuff too." It's depressing.

* or substitute whatever other enterprise framework you want

all of them really mean "Hadoop' babysitter with a dash of full-stack

...because that's all that most business require, 99% of the time. To you know, get things done and make money and stuff.

Which may not fit your needs, but why be "depressed" about it? It's just the way things are in the commercial world.

If you want someone with more fine-grained stills, try articulating that in your job postings. What we see, all the time, are ads mentioning platform X, with no articulation whatsoever as to where, even on some approximate logarithmic skill, they'd like the skill level and comfort with platform X to be.

That's rarely why these enterprise frameworks are bought or operated. More often it's for various permutations of the "no one got fired for buying IBM" excuse. Big showy re-orgs around Hadoop are mostly for status effects, hardly ever related to engineering realities.

And in the few firms that actually do have real engineering trade-offs that favor the use of those types of frameworks, they tend to hire people who are well-suited for the role, and then create job functions surrounding them that are respectful of aptitudes and skills of the people they hire.

In most firms that adopt these frameworks (for status effects), they are just desperate to fill seats and increase engineering headcount. They don't respect your skill set or even care if it matches the business need. They just need to get you in the door, and then find a way to deal with inevitable dissatisfaction later.

You didn't get picked for the special snowflake job that you trained for then and are made to do grunt work instead?

Welcome to the real world.

Do you think a decent full stack engineer would have a problem picking up "proper" data science on the job? Both are fairly intelligent jobs with a fair bit of overlap.

What most firms mean by "data science" is basically just using tools for data cleaning, visualization, and using APIs like black boxes for a few different kinds of models. You are often not even allowed to take a software design approach to these tasks, and often you're just writing scripts in a hurry to address "business intelligence" fire drills.

For those positions, I am sure that smart full-stack developers could easily pick up the statistics for data cleaning and quickly gain a passable understanding of the models consumed from APIs in a black box way. In fact, full-stack devs may be happier in these jobs due to the visualization and database components.

A much smaller subset of data science is actually focused on solving novel business problems and may centrally focus on deeper knowledge of a given technique, like MCMC methods, deep learning, real-time classifier systems, etc. For these, you do tend to need more significant experience with the specific machine learning tools being used (or enough general skill in statistics to pick them up quickly). Smart people of all stripes could still learn that stuff, but it's a lot harder to see them being able to convince a firm to hire them in that capacity.

The second type of these jobs is really, really rare though.

Clearly this is a failure of LinkedIn's ML team to properly align people with usefully overlapping needs. The dark patterns have failed us both. :-)

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact