
Erlang at Basho, Five Years Later - bsg75
http://basho.com/erlang-at-basho-five-years-later/
======
seldo
I think the most important point here is that you should hire engineers from
_your domain_ , but not necessarily your exact technology stack -- in their
case it's simple because almost no engineers know Erlang well, so they have to
hire "database people" and convert them, but I think it's more generally
applicable.

Personally, I would much prefer to hire a "web person" regardless of the stack
they're used to than, say, a "Java person" who has never written a website.
But even more specifically, I would look to find somebody whose interests are
in my domain, as they will always have better ideas and motivation than a
strictly mercenary engineer.

~~~
smegel
Not to mention the fact that most experts will spend their time on concept on
design rather than code cutting. I think Rob Pike said it well when he said he
didn't event have commit privileges at Google while working on Go because he
still coded in K&R C.

------
davidw
It's a pity they didn't talk about Erlang's strengths and weaknesses in a bit
more detail.

I like Erlang a lot, and pushed to use it in a project I'm working on - it is
the right tool for the job in this case.

But programming languages aren't like hammers or saws. You can do a ton of
things with any one of them. This particular project probably could have been
made to work one way or the other with Java, Javascript, Tcl, Ruby, Python,
Perl, PHP, Haskell or any number of other things. And this is, in some ways, a
problem for Erlang. It's really, really good at a few things, ok at a number
of things, but feels a bit spotty for other things where other languages have
something in place. At least that is my impression.

~~~
granitepail
I saw the article as more of a reaction to all of the technical (and often
times inane) blog posts about why company x uses technology y.

The main points I took away from the article were:

\- They did their homework, and \- They knew that, even though it is not the
most widely known technology stack, they were seeking the kind of engineer who
would not only be able to pick it up, but understand their rational for
choosing it

And it seems as though they've been successful! As engineers, most of us are
interested in the many interesting technologies available to us, so much so
that, when examining a company like Basho, many of us ask "Why Erlang?" before
we ask "What are you making and why?"

Without putting words in their mouth, it is my impression that Basho sought to
create an incredibly powerful and easily scalable database. They chose Erlang
because it was, in their opinion, the tool for the job. As I see it, 10gen got
the ball rolling towards approachable scaling/development and now Basho and
the team behind RethinkDB are trying to improve by creating systems free of
MongoDB's many limitations.

~~~
leif
What do you see as MongoDB's limitations, and are any/most of them solved by
(the product I work on,) TokuMX[1]?

[1]: [http://www.tokutek.com/products/tokumx-for-
mongodb/](http://www.tokutek.com/products/tokumx-for-mongodb/)

~~~
granitepail
You guys seem like a step in the right direction! The improved locking, in
particular. Sharding is still a nasty issue w/ MongoDB.

------
rch
This is a phenomena worth understanding, if you don't already:

"I had an entertaining and ironic conversation about this recently with a
manager at a large database company. He explained to me that we had clearly
made the wrong choice, and that we should have chosen Java (like his team) in
order to expand the recruiting pool. Then, without breaking a stride, he asked
if I could send any candidates his way, to fill his gaps in finding talented
people."

------
xaritas
I found Erlang to be much easier to digest the second time through. The first
time I looked at Erlang, I knew of its benefits, but didn't have much
experience with functional programming. The unusual syntax and unfamiliar
paradigm led me to put it aside.

Later, after I learned me some foldr and flatMaps (mostly thanks to Martin
Odersky's Scala/FP course on Coursera), I revisited the language when I saw an
article here about an Erlang based CMS
([http://zotonic.com/](http://zotonic.com/)). It seemed pretty cool, I host
some websites for local businesses, so I set up it to evaluate it and realized
that I now had no trouble at all with Erlang and the syntax made more sense.
Searching around for some other web stuff I found Chicago Boss
([http://chicagoboss.org/](http://chicagoboss.org/)), a Rails-inspired web
framework with ridiculously easy Comet/WebSocket/"real time" support.

Now I'm really excited about it and I can't wait to do some cool things with
it.

~~~
aeden
My experience was quite similar, although it took me three tries to get it. I
think it was [http://learnyousomeerlang.com](http://learnyousomeerlang.com)
that finally helped me understand enough to really start to enjoy Erlang.

------
fusiongyro
The large database company example is especially telling. The company I work
for chose Java mainly for the benefit of a large talent pool, but we have
great difficulty hiring people. I think it's because we're in the middle of
nowhere and don't pay Silicon Valley wages. I think if we had chosen a more
"offbeat" language (to borrow mosburger's adjective) we would have had a
novelty factor that would increase interest in the job. After all, we're doing
interesting stuff, but our technology decisions make us seem very dull.

 _Edit_ : Please note I don't have the authority to make a technology change
and see what the effect would be on our hiring, despite the good ideas of
those replying to me.

~~~
geoka9
You could start migrating to Scala, which is simple with a Java-based stack.
Scala is sufficiently hip at the moment to attract programming enthusiasts.

~~~
mixedbit
Similarly Clojure is JVM based and it draws interest of many Java gurus,
because they do not need to learn the platform from scratch (the Java library
is available).

~~~
JPKab
I think you mean Clojure.

~~~
mixedbit
thanks, fixed

------
mosburger
This is sort of similar to pg's "Python Paradox" essay.
[http://www.paulgraham.com/pypar.html](http://www.paulgraham.com/pypar.html)

It's a slightly different conclusion (use best tool for the job, earn respect
of developers vs. use esoteric tool for the job, attract developers eager to
learn something off-the beaten-path), but with similar ends. Using an offbeat
language can have its benefits.

------
benmmurphy
it would be interesting if they covered some of the technical problems they've
had with using erlang in a database. writing a db in a VM language like erlang
or java you inevitably end up fighting with the VM for some workloads. i
noticed that recently one of their engineers tried to submit a patch to erlang
to force it to not put schedulers to sleep when there is work to be done
([http://erlang.org/pipermail/erlang-
patches/2013-June/004109....](http://erlang.org/pipermail/erlang-
patches/2013-June/004109.html)). it had an interesting flag name '+zdnfgtse'
which looks like z-do-not-f-go-to-sleep-ever :)

------
roncohen
I do believe that Erlang is the right tool for the job when you're building a
distributed database, especially for the _distribution_ part. For a company
like Basho it's understandably a good investment to take expert distributed
systems/database engineers and let them take their time learning Erlang.

If, however, you, like most of us, are not building a distributed database,
think carefully before you choose Erlang, especially if you're in a startup.

Most startups don't have a distributed database as their core technology, and
for many startups, it's important that developers can push production code
from day one.

~~~
derefr
I think the important question for HN, here, is: what "app ideas" map, in the
end, to "having a distributed database as core technology"?

If you're building the next Facebook, is that, fundamentally, a distributed
database?

If you're building an MMO, is that, fundamentally, a distributed database?

If you're building a payments platform, is that, fundamentally, a distributed
database?

And so forth.

~~~
oneweekwonder
What distributed application can not be "philosofed" into essentially being a
distributed database, and I feel like I'm just repeating your question.

With that in mind, I believe it boils down to where is the logic located, with
the data, or not. Because a lot of "databases" are just used as a datastore.

So let me try to create some scenarios:

1\. To push out a product(Startup/Prototype/POC): Develop in the technology
you know, and let the stake holders know you are acquiring Technical Debt.
After startup/prototype product is a success. Explain to stake holders why you
need to move to a better technology stack that supports distributed systems
"better".

2\. If you have a design or product, develop directly in a distributed
technology stack.

It boils down to use the correct tool for the correct job, but you will mostly
only know if you used the correct tool later in the development of the
product.

------
josh2600
We use Erlang at 2600hz and I feel like this is an Echo Chamber of the kinds
of conversations we have around the table. Erlang is a beast, but if you're
doing hard work it's more about the idea than the tools.

Lots of wisdom on this blog post. The truth is that if you're passionate about
doing something hard, and communicate that passion, people are drawn to it.

------
cinbun8
I like that they focus on the experience related to DBs and distributed
systems instead of hiring engineers that can program in X. You can pick up a
language relatively faster since you'll need to build the other skill slowly
with real world experience.

------
bsg75
"While it’s theoretically a nice bonus for someone to bring knowledge of all
the tools we use, we’ve hired a significant number of engineers that had no
prior Erlang experience and they’ve worked out well."

Nice to hear a company understand that talent and abiliaty does not equate to
n-years using a specific toolset ('not “expert hammer wielder”').

------
outworlder
The only thing I didn't like about Erlang is its syntax - but I am sure I
could adapt. I just don't like too much syntax in general, one reason why I
like Scheme.

Now, Erlang's runtime seems fantastic.

I wonder if they (Basho) investigated Elixir recently and their opinion of it.

~~~
seancribbs
Elixir has some good things about it (hygienic macros, protocols, and so on)
but the syntax is not enough by itself to make us switch. "Ugh, Erlang syntax
sucks" is a refrain heard around the Internet, but it has not hindered us from
getting good engineers who are happy to develop with it. I daresay that the
syntax is the _least_ of your problems when learning Erlang; the message-
passing concurrency and fault-tolerance models, and functional style are more
difficult to understand. Those problems don't go away when you use Elixir
instead.

If you want something Scheme-like on the Erlang runtime, there's LFE (Lisp-
Flavored Erlang):
[https://github.com/rvirding/lfe](https://github.com/rvirding/lfe)

~~~
rdtsc
> I daresay that the syntax is the least of your problems when learning
> Erlang; the message-passing concurrency and fault-tolerance models, and
> functional style are more difficult to understand.

That's my feel as well. I hear tirades about commas and periods. What are they
going to do when they hit a net-split.

Bashing Erlang and stopping at syntax is like bashing a new battle tank
because it doesn't have a leather interior. Yeah it is nice if it had leather
seats and a mini fridge but if that is really the main criteria used in
picking it, one has to wonder...

------
peteretep
Do other people have much experience cross-training people? I am struggling to
find decent Senior Perl developers at the moment in London. I am open to the
idea of hiring a Senior X Dev and cross-training them ... I'm just not sure
how well it'd work out.

~~~
JanezStupar
I would rather have a wise and experienced developer X than a developer with Y
experience in Z.

As long as you are hiring people with skills in solving similar problems to
the ones you are having or are anticipating on having.

Edit: However that may not work for you. I think that a senior Perl developer
would be more than competent in any language you point towards. While the
opposite may hardly hold true. The presumption is that one believes that a
senior Perl developer really exists.

