Indeed. The only recent Scala job's I have seen in Australia have been Spark jobs because Spark uses Scala. Even these are diapering for Python Spark roles.
I've worked on a few big data projects for large companies. I have a huge dislike for them as mostly seem about how can we fuck over a customer to better the business or do shady stuff with the data rather than anything meaningful.
That and it's mostly just ETL, aggregating various data sources, cleaning, putting the data in some central location then running queries on it and the "If the only tool you have is a hammer, you will start treating all your problems like a nail." problem, tiny data sets going from NiFi / Hadoop / etc when `cat ... | awk ...` will be more than sufficient.
Yes for us Spark is being used for just ETL. It's internal data though. It's at the scale where no single machine could ever handle it though. It's replacing large proprietary systems like mainframes and teradata.
We wrote a framework to heavily simplify writing these ETL flows. No developer of the ETL flows is using Scala unless something custom is required. I wish we could opensource it.
>I've worked on a few big data projects for large companies. I have a huge dislike for them as mostly seem about how can we fuck over a customer to better the business or do shady stuff with the data rather than anything meaningful.
I've worked on a few big data projects for large companies. I have a huge dislike for them as mostly seem about how can we fuck over a customer to better the business or do shady stuff with the data rather than anything meaningful.
That and it's mostly just ETL, aggregating various data sources, cleaning, putting the data in some central location then running queries on it and the "If the only tool you have is a hammer, you will start treating all your problems like a nail." problem, tiny data sets going from NiFi / Hadoop / etc when `cat ... | awk ...` will be more than sufficient.