
Spark killed Hadoop - panchicore3
https://www.datanami.com/2017/09/29/hadoop-hard-find-strata-week/
======
lukaslalinsky
The title should be changed. The article specifically ends with saying that
Hadoop is not dead. Spark might have killed MapReduce, but the rest of Hadoop
is alive and very useful. The only alternatives worth considering for
replacing those parts of Hadoop are either proprietary or too dependant on a
single company.

------
chrispeel
...and HiFrames [1] and HPAT [2] will kill Spark ;-)

HiFrames and HPAT are built on Julia [3] and use the Julia packages
ParallelAccelerator.jl [4].

[1] [https://arxiv.org/abs/1704.02341](https://arxiv.org/abs/1704.02341)

[2]
[https://github.com/IntelLabs/HPAT.jl](https://github.com/IntelLabs/HPAT.jl)

[3] [https://julialang.org/](https://julialang.org/)

[4]
[https://github.com/IntelLabs/ParallelAccelerator.jl](https://github.com/IntelLabs/ParallelAccelerator.jl)

------
ganeshkrishnan
Spark still uses Hadoop underneath. While Hadoop mapreduce uses the disk,
spark uses memory for faster processing. And I am not sure anything is going
to kill spark soon. Once a library gains critical mass, it's harder to replace
it in existing systems

~~~
doug1001
Spark can read from and write to HDFS, and YARN (the resource negotiator in
Hadoop v 2) is one of the cluster managers supported by Spark.

but you can run a Spark cluster without YARN--eg, with Mesos--or with the
built-in manager provided in the Spark distro. Likewise, your Spark cluster
doesn't need to read or write to HDFS.

