
Ask HN: Is there a Hadoop alternative built on non-Java technologies? - tmbsundar
For distributed or &quot;big data&quot; scale computation Hadoop has been so successful that, one is not able to find information about any alternatives that exist. For one, it will be good if any of the ML family of languages&#x2F; LISP etc., had an viable alternative. There are Python bindings that exist and DSLs like Pig&#x2F;Hive simplify things a bit. But beyond that, without knowledge of Java it&#x27;s hard to navigae the Hadoop ecosystem. Hence this question.
======
mindcrime
It depends on how tightly you define "Hadoop alternative" and exactly what
you're trying to do. But generally speaking, the answer is "yes". There are
technologies for doing large-scale distributed computation on clusters of
commodity computers that don't require Java. In fact, many (most?) of them
predate Java. You can, for example, build a Beowulf cluster[1] using something
like Rocks[2] and program using MPI[3], OpenMP[4] or PVM[5].

The one big difference is that you don't get the advantage of the (relatively
speaking) easy to understand and code against Map-Reduce API. Programming in
MPI requires a bit more low-level fiddling... something like the difference
between, say, programming in C vs programming in Java. Which is, I suppose,
quite appropriate since most MPI programming is done in C (or using the C
bindings from another language).

There are also some Map-Reduce implementations natively written in C++
floating around out there.

Note that if you do something like a Beowulf cluster you still need a
distributed file-system to serve the role HDFS plays in the Hadoop world.

There's also this:

[https://gigaom.com/2015/02/18/google-open-sources-a-
mapreduc...](https://gigaom.com/2015/02/18/google-open-sources-a-mapreduce-
framework-for-c/)

but I haven't used it and don't know much about it.

[1]:[https://en.wikipedia.org/wiki/Beowulf_cluster](https://en.wikipedia.org/wiki/Beowulf_cluster)

[2]:
[http://www.rocksclusters.org/wordpress/](http://www.rocksclusters.org/wordpress/)

[3]:
[https://en.wikipedia.org/wiki/Message_Passing_Interface](https://en.wikipedia.org/wiki/Message_Passing_Interface)

[4]:
[https://en.wikipedia.org/wiki/OpenMP](https://en.wikipedia.org/wiki/OpenMP)

[5]:
[https://en.wikipedia.org/wiki/Parallel_Virtual_Machine](https://en.wikipedia.org/wiki/Parallel_Virtual_Machine)

