
Scaling Erlang to run on machines with hundreds of thousands of cores - adg001
http://www.erlang-solutions.com/news/1/entry/1277
======
karterk
Direct link to the project:

<http://www.release-project.eu/>

Synopsis of the project:

<http://www.macs.hw.ac.uk/~trinder/RELEASEfactsheet.pdf>

------
gtani
" near-perfect scalability of the Big Bang benchmark on a simulated 128-core
SPARC"

[http://erlang.org/pipermail/erlang-
questions/2011-July/06029...](http://erlang.org/pipermail/erlang-
questions/2011-July/060297.html)

Also, sounds similar to grant to Scala team, Jan 11

<http://www.scala-lang.org/node/8579>

------
tsotha
How big is the demand for applications that run on hundreds of thousands of
cores? I can think of a few things, like weather prediction, but is this
really a sizable market?

------
wmf
The description is a little confusing; they're talking about clusters, not
really machines.

~~~
nirvana
In erlang there is not necessarily any difference between a machine and a
core.

If you have a CPU with 100,000 cores, then you could run the same erlang
program on it that you run on a cluster with 100,000 machines with the only
changes being addressing the nodes on the cores rather than over the network.

~~~
pjscott
No difference in the programming model, perhaps, but the internals are very
different.

------
nirvana
Erlang is the future, and the past of concurrency. Any developer worth their
salt really should be learning it. If you will take a bit of time to do so,
you'll discover that rather than the incomprehensible scary pain you expect
when you first see it, it is quite logical, and quite a joy to work with.

In fact, it is a litmus test for me. When I first started looking at NoSQL
solutions, I quickly heard of MongoDB. When I found out it was written in C, I
suspected that it wouldn't be distributed and it wouldn't be concurrent... and
in researching further, those suspicions came true. (by "distributed" I mean,
homogeneously like Riak, not the brittle master-server-sharding setup.)

Trying to write a truly distributed, concurrent application in C is an
exercise in frustration-- you'll have to produce halfway done poor
implementation of erlang to do it, or you'll deal with pain every single day.

But if you start with erlang, you get these features essentially for free.
Plus a couple decades of seriously engineered OTP to go with it.

I know a lot of "developers" these days never use anything more ....
intense.... than a scripting language. But if you're ready to dive into the
deep end, it isn't replicating the experience of writing assembly in C, it is
going towards the future-- erlang.

~~~
mxavier
I'm fairly new when it comes to working with Erlang. We've developed an
application we use internally for essentially doing distributed, fault-
tolerant timers.

I, however, would caution against some of the platitudes you use about it
being the future of concurrency. I also think it is kind of silly to imply
that a coder isn't "worth their salt" if they aren't learning Erlang. It isn't
a panacea. If you read some modern Erlang guides like Learn You Some Erlang,
for example, you'll note that a responsible advocate of Erlang will caution
against drinking too much kool-aid. Erlang, for example, is admitted as being
a poor choice for lots of string manipulation because of the way strings are
implemented in the language. I believe that same guide mentions that it may
not be the best choice for heavy number crunching.

Developers worth their salt will not just hear the word concurrency and start
writing everything in Erlang. It is much better to consider the problem and
ensure that the problem domain matches Erlang's particular strengths.

I also want to mention that in my experience, Erlang applications are pure
hell to deploy for people who don't have years of OTP deployment under their
belts. So much of having a manageable Erlang app depends on the developers
being well versed in a lot of subtle, and very easy to get wrong conventions.
We actually ended up needing to start a support contract with Erlang Solutions
themselves to help us demystify getting a sane deployment process going for
our app. This was an expensive endeavor. Yes, it was primarily because we were
new to the language, but it also had a lot to do with a lack of community
documentation/discussion about deployment procedures. I won't even get into
the joy of trying to decipher Erlang's famously cryptic crash reports/stack
traces.

tl;dr: Erlang isn't concurrency pixie dust. A developer isn't a failure if
they aren't currently learning it, nor are they a success if they just decide
to use Erlang for every class of problem. It is an interesting language to
work with and solves certain problems very well, others not so well.

~~~
nirvana
If you're not using erlang because you don't like its string manipulation
capabilities, then you're being foolish.

If performance is critical, you can do the string manipulation in C and plug
it into erlang fine.

If you're trying to do concurrency in most other languages, then you've
probably made a foolish decision, unless those languages are also message
passing actor model.

Claiming that deploying erlang is pure hell is FUD, or an example of how
you're doing it wrong and really shouldn't be commenting on erlang.

~~~
runT1ME
I'm not convinced the Actor model is a concurrency silver bullet. I'm using it
in Scala, and you still have race conditions and potential deadlocks to worry
about.

It helps to 'segregate' your application so it's easier to reason about what
data could be a race sometimes, but for me it just made multithreaded
applications a little less hard, not easy.

~~~
jerf
The way I like to put it is that it takes concurrency from an exponential
problem that no human can actually solve, to a polynomial one. It isn't a
magic bullet, because nothing can truly fully abstract away concurrency, but
at least it's in the class of sane answers.

This is grounded in the arguments about how many paths through your program
there are; with conventional threading, it's exponential since at any time any
thread may reach out and twiddle with something another thread has. Things
that isolate threads confine interactions to just their communication points,
which is more polynomial than exponential. You can still get yourself in
trouble, but at least you don't start out in trouble.

~~~
Retric
A secret of rock solid concurrency is communication, but there are many sane
concurrency models. Consider people are already fast efficient safe sane code
over hundreds of cores on GPU's which looks nothing like Erlang. Still, when
you actually start to care that your concurrent code is accurate not just FAST
you need to validate your calculations and Erlang does little to help you
there. So sure it's better than C, but turn up the memory errors and it still
falls down hard.

