Building or running data infrastructure is an important part of 55% of 372 data engineers’ jobs, according to the “2020 Kaggle Machine Learning & Data Science Survey.” ...Check out three charts I created showing differences between data scientists, data engineers and machine learning engineers: https://thenewstack.io/software-engineers-use-spreadsheets-d...
Monoliths are "part of the future". Data shows that there is dissatisfaction with using both monolithic and microservices architectures at the same time. In 2017, 66% were using hybrid approach to approach, but only 54% in 2019. See the interactive graphic at the bottom of https://thenewstack.io/observability-and-elk/.
Yes, this was a big job -- one of the reasons I stopped working on something similar 1.5 years ago was that I wasn't capturing the non-GitHub activity. Of course that back then I also wasn't controlling for bots.
Just checked, the # of data engineers rose to 9,246 (42%) in the last six months. So, the shortage is at least being addressed by people changing their job titles on LinkedIn.
