Hacker News new | past | comments | ask | show | jobs | submit login

Title should have been "Engineers Shouldn't only write ETL." I agree with the author's statement of the problem, but not with the proposed solution. Succinctly, I think the problem is compartmentalization and specialization. These are qualities that are sometimes promoted by management so that it is easier to maintain control over the organization and to hire people who won't require much training to do their jobs. Unfortunately, compartmentalization and specialization both lead to unhappiness in the workers, and are net negative for production. I believe the solution is fostering a wholistic approach among the specialists. Data scientists (who should be statisticians or machine learning experts) should interact regularly with software development engineers that have to productionize their research and they should also both interact regularly with systems and database administrators who make it all work in production. Rather than being separate teams working on parts of the same goal, they should all be one team. By working together through the poroblems faced in each area, they can learn more about each other's areas of expertise and will create a better solution faster. This isn't true just for data science, but throughout technology, where operational software developers should work together with product development, marketing, testing and operations to break down the divide and get all team members working towards the same goal.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact