
Ask HN: What have you built with Erlang? - SnootyMonkey
A recent Erlang article in Communications of the ACM portrays Erlang as a specialized language for messaging/chat (Facebook chat, XMPP, etc.)and distributed storage (CouchDB, Riak, etc.)<p>I think this is probably right, but I'm wondering what else you've built in Erlang. Was it a good fit? Why or why not?
======
KirinDave
I built a fault-tolerant system for binding lots of disparate and network-
unaware services via a monitored event-driven communication protocol to a
network-fronted cloud architecture. It's called Katamari, and it's the version
of Fuzed (this same concept just for Rails instances) that I modified to run
as the majordomo of Powerset's realtime query analysis layer. It let us take
un-threaded and network-unaware software that talks over local file
descriptors and turn them into a cloud service.

Katamari also has some fairly good fault tolerance characteristics. It
aggressively tries to maintain a cloud and it has no real single points of
failure. It recognizes hung or crashed instances of individual workers and
restarts them. It also supports arbitrary versioned resources existing in
parallel, which is great when you want to do A/B testing.

It was also designed so that it was usable in both a development and
production environment. While currently little slow for most production uses
(~2ms overhead for most queries, with a special case of ~12ms), it's plenty
fast and runs in less than 10mb of memory on most linux systems, and can
transparently go from a 1-box to a N-box solution.

Microsoft open-sourced it for me (after 2 years of bureaucratic nonsense), so
I am just pulling out the useless and crufty parts before I put it up on
GitHub. I've given 2 talks about the technology, one for Fuzed at Railsconf
and one on Katamari at the last Erlang Factory. I hope to have a modernized
version maintained this year, and start moving the tech forward.

Katamari is about 10k lines of code; with about 2/3 of that being Erlang and
the remaining 1/3 being Ruby code for the glue (Ruby was Powerset's primary
glue and binding language, so it is the only supported binding). Katamari
aggressively leverages the characteristics of Erlang to accomplish this
feature set; its safe to say all the fault tolerance wouldn't have been doable
without Erlang's implementation providing the basics for it.

------
EvanMiller
ErlyDTL, a Django Template Language compiler:
<http://github.com/evanmiller/erlydtl>

Chicago Boss, an MVC web framework in progress:
<http://github.com/evanmiller/ChicagoBoss>

This doesn't get mentioned enough, but Erlang has excellent compiler tools, so
if you're willing to roll up your sleeves, you can usually hack the language
to fit your problem. For instance, Chicago Boss ships with a nice but
completely non-standard database querying syntax, described here:

<http://www.chicagoboss.org/api-db.html>

You will weep for joy when you rewrite a gnarly SQL query with a few well-
chosen Unicode set operators.

The compiler tools also make it easy to generate Erlang AST and compile it to
BEAM (Erlang bytecode). That's how ErlyDTL works. I haven't done performance
benchmarks myself, but the folks at Zotonic report that their Django templates
and controller code come out 10X faster than comparable PHP code.

I think a couple of things prevent Erlang from being a do-all language:

1\. Lack of libraries. The situation has been steadily improving, but Erlang's
VM and processing model makes it difficult to draw upon the "C tradition" in
the same way that Perl, Ruby, and the others can take a popular C library and
write a thin wrapper around it.

2\. Inferior OS integration, at least with UNIX. For instance, custom signal-
handling facilities are non-existent, and you have to jump through strange
hoops to do simple stuff like close only the write end of a read-write pipe,
or get the exit code of an exec'd process. Weak sauce for anyone who has spent
a lot of time with Perl, Ruby, or C.

------
yariv
ErlyWeb, an open source web framework: <http://github.com/yariv/erlyweb/>.

Twoorl, an open source Twitter clone: <http://github.com/yariv/twoorl/>.

Vimagi, a graffiti-like facebook app: <http://apps.facebook.com/vimagi-paint>.

Also, parts of Facebook chat (I work at Facebook).

~~~
ak1394
Also, thanks for your work on mysql driver, which I happen to use for PavoMe.

~~~
adelevie
HN is such a nice community :)

------
KevinMS
I built a very specialized mail server (MTA).

It was a tedious, frustrating experience, even though you'd think it would be
the perfect language.

It ended up being too slow (probably because of all the string processing)

In my entire programming career, I've never before had to join a mailing list
and ask questions to get things done, that should tell you something.

~~~
tptacek
I'd be interested in hearing more about the frustrations.

~~~
keefe
yes indeed I'm increasingly using couchdb, planning on picking up erlang and
it would be useful to avoid the mailing list if you can pass info along...

~~~
KevinMS
This was a few years ago, so my memory is foggy and things might have changed,
but I remember how uncomfortable erlang felt in a unix environment. I might
have felt the same way about java, since its also sandboxy.

I think the first time I hit the mailing list was because I couldn't figure
out how erlang can boot up and change its effective user id. (setuid). The
reason being it needs to grab a privleged port as root. Well forget that, it
has no idea what that means. The recommended way is to use iptables port
forwarding. I think thats how couchdb does it (or one of those nosql things)

Logging also didn't seem to fit well with linux. You can manage to write some
stuff to syslog, but erlang really prefers its confusing way of logging. See
things like this <http://www.erlang.org/doc/man/sasl_app.html>.

Oh and if you decide to just write data to the error log in the simpliest way
possible, like this error_logger:error_msg("An error occurred in ~p~n",
[a_module]). Guess what, it doesn't open that log file in append mode, so it
wipes it out every time you restart the erlang process. You have to write and
install your own error log handler to open the file in append mode.

Want to do some dns lookups? Sounds simple. It starts up a gen_server
thread... and starts caching the results! Talk about principle of most
surprise. And it doesn't handle mx records. But if you get on the mailing list
the guys will tell you there's an undocumented library deep in the distro that
will handle this, and probably won't change. So you give up in frustration and
just write a port to run dig.

It just went on and on.

You can really tell that erlang evolved to run self sufficient on special
hardware and not in a supportive unix environment.

And of course there are the basic language frustrations. What bothered me the
most was simply the lack of a 'return'. This meant you couldn't 'short
circuit' a function, like "if something then return", and it often resulted in
another level of nesting, even for the simpliest stuff. Yes I know that
sometimes you could use function pattern matching to factor this out, but not
as much as you hope. Having a 'return' statement in erlang is part of a
religious war, which I'm not going to fight, but I'm strongly on the having
side.

I found some erlang things worked great for my project, like the server and
finite state machine 'behaviors', but when I realized the string processing
was so slow I just switched to stackless python, wrote it in a fraction of the
time, never used it, and ended up keeping the existing perl version :)

You can write a MTA in any language, so its probably not the best fit for
erlang. The best fit for erlang would be something that only erlang could do,
and just suffer through it ;-)

~~~
ams6110
_You can really tell that erlang evolved to run self sufficient on special
hardware and not in a supportive unix environment._

Well yeah, it was written to run telephony equipment. It's not a general
purpose language, though it has been moving that way.

~~~
acqq
AFAIK it was written as the research, not to run at any specialized hardware,
then "sold" as being able to run the phone switch, but it needed a "big iron"
computer attached to it just to drive that switch (!) -- it was _that_ slow.
And AFAIK it's still very slow. Up to quite recently it wasn't even able to
use more cores.

~~~
rvirding
Where did you get all that from? Erlang was never intended for research but as
a tool for writing real switch applications. Yes, the original Prolog
interpreter was slow, but that has not been used for almost 20 yrs. We never
had "big iron" computers to play with, and never really needed them either as
the system has been efficient enough to run on computers that were cost
effective for the actual applications. The implementations have always been
optimised for applications that are highly concurrent and extremely fault-
tolerant, like telephone switches. Yes, it was a bit slow in being able to use
more cores but that was because Ericsson, who still maintains the BEAM
implementation, did not have need of it. Now, it will take as many cores as
you can throw at it, and be able to use them if you have written a truly
concurrent application (Amdahl's law). Transparently too.

~~~
acqq
I remember reading "Erlang - A survey of the language and its industrial
applications" (Joe Armstrong - in INAP'96)

Now I've checked again, so let me summarize:

It started to spread "outside the lab" in 1987. In 1995 there were three
commercial applications: network simulator, mobility server and distributed
resource controller. First is a simulator, not running on the specialized
hardware, the third was running on "a network of Windows NT machines." The
second: "acts as a call control system _when linked to the PBX_ "

OK, the second was running on the additional SPARC-based machine, not big
iron, but AFAIK certainly more powerful than the PC's of that time.

So I don't see anything that was targeting really "specialized hardware"
particularly not what we'd think as embedded (less power consumption, slower
processors than the PC CPU's). It was a research project, and it had
traditionally performance problems.

Even today, after so many years, when you compare Erlang HiPe (Hi Perf, you
can imagine others then!) with something higher than C, let's then say Lua JIT
it's easy to see which one is faster:

[http://shootout.alioth.debian.org/u32/benchmark.php?test=all...](http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=hipe&lang2=luajit)

Edit: Or for multi core 64 bits with Mono:

[http://shootout.alioth.debian.org/u64q/benchmark.php?test=al...](http://shootout.alioth.debian.org/u64q/benchmark.php?test=all&lang=hipe&lang2=csharp)

~~~
evgen
Do you realize that rvirding is Robert Virding, one of the creators of Erlang?

------
hassy
I worked on a spreadsheet compiler at hypernumbers.com and some backend stuff
for vidiowiki.com (video transcoding, file uploading/streaming and some other
smaller stuff).

Erlang excels at distribution. People tend to forget its raison d'être is
fault-tolerance (not concurrency), so when you set off working on an Erlang
application you presume that it will run on 2+ machines. That's why it's great
for things like Facebook Chat and Riak. In addition to those, it's also good
for writing control and supervision layers, where workers are written in
Python, Java, C etc.

While I'm at it, I'd like to plug <http://veldstra.org/whyerlang/> It's in a
real need of an update, and I'd love to hear any feedback.

------
francescoc
Banking systems, both retail and commercial. Logistics. Embedded. Clusterware.
Telecoms. Robotics, AI. When it comes to messaging, don't forget RabbitMQ and
all the SMS systems.

Francesco

~~~
sgt
For those of you who don't now, Francesco Cesarini is the founder and CTO of
Erlang Training and Consulting, and has been a major proponent of Erlang for
years, also having co-written the book "Erlang Programming"
(<http://oreilly.com/catalog/9780596518189> \- it's a great book, I highly
recommend it myself)

------
ngerakines
A Facebook application that had nearly 2 million users.

A game data aggregation platform and feed system for EA.

A lot of open source Erlang stuff (<http://github.com/ngerakines>)

------
ovidiu
I have built the backend (game logic and comet) for
<http://www.iluminatus.ro>, a trivia-like multiplayer game for the Romanian
market. We're going to release an english version soon.

It was a great fit. It lacks Unicode processing capabilities by default, but
there are a few open source libraries for that. The only thing I dislike about
Erlang is the poor string manipulation and lack of clean structures (Erlang's
records are an ugly hack IMO).

------
gordonguthrie
<http://hypernumbers.com> \- an online spreadsheet which is a native web app -
lurching towards a proper launch after a lot of development.

Erlang is a great fit - works well, allows you to build a native stack without
any cross-language impedance. Your application is cluster ready from the get-
go and allows you to focus on the functionality rather than any of the
utilities required to make the damn thing stay up.

------
ak1394
I built a Twitter client, with j2me on handsets and Erlang server at the back
to handle all communications with twitter and few other services for image
hosting, etc. It was my second project in Erlang and was very rewarding
experience. Part I especially liked, is that I could deploy updates to the
server while it was running, with no desruption to clients at all.
<http://pavo.me/>

------
BrandonSmith
Phonebooth, a hosted telephone system <http://www.phonebooth.com>

Great fit. OTP process restart characteristics. Per-tenant process separation.
Process abstraction, in general, rather than threads. Finite state machine
behaviors. Live code upgrades. Mnesia database.

Only real downside was it took the rest of the team a bit of time to become
effective in Erlang... about 4 to 6 months.

------
sofuture
Simple stuff, but:

\- An IRC bot -- great fit for the language

\- A URL shortener service (three times) -- first on Yaws, then on Nitrogen,
then on ChicagoBoss (Hi EvanMiller!)

Erlang is the most fun I've ever had writing code. While a lot of that is the
functional language aspect, I find it an extremely intuitive language at least
at a low level (once you get past the 'there are 4 ways to end a line?!?!'
stage). I'm still getting used to thinking through bigger problems in an
Erlang way, but thats more a function of my brain than the language.

I think Erlang gets a lot of unfair knocks given it's pigeonholed reputation
for 'concurrency and fault tolerance'. Certainly it has it's limitations
(libraries), but generally, I think its a pretty well-rounded general purpose
language.

------
samuelth
Not me but deserves attention: <http://www.wings3d.com/>

"Wings 3D is a subdivision modeller with an user interface that is easy to use
for both beginners and advanced users"

------
fmstephe
I wrote a distributed sat-solver in Erlang (good old fashioned search
problem). It was my intro project to get to know the language. Has been great
fun and now Erlang is in charge of computers talking to each other while Java
takes care of the heavy lifting. Also a similar problem solved with Erlang
(nothing to do with me), and featured here on hacker news...

[http://blog.ezyang.com/2010/08/tour-of-preach-distributed-
er...](http://blog.ezyang.com/2010/08/tour-of-preach-distributed-erlang/)

Great stuff. Erlang may make you smile (again).

------
bnicholso
We built a system to connect teams via texting and conferencing.
<http://www.connectteams.com>. Erlang provides the web front end, the backend
telecommuncations, and the reporting interfaces. All perfect fits.

------
vito
Super late (a misjudged apache-bench got me banned for a day or something),
but: <http://darcsden.com/alex/aliter>

It's a reverse engineering of a popular MMORPG called Ragnarok Online (or RO).
Last time I was working on it, you could log in with the official client, walk
around, teleport, chat, etc. (The walking had to be done in C, though; one
pathfind per walk request is not something I'd want to leave to Erlang.)

------
gleber
Erlang is perfect for any messaging systems. And on other hand Erlang works
well for GIS:

Vector Map Server - a BSc thesis project. Vector geospatial data streaming
server, very similar thing to server's streaming vector data to Google Earth
or Map24. Also works as WFS server. Has to handle a lot of data, but since
data is mostly immutable it works well.

Backend service for not yet released liveblogging site. Done in collaboration
with few other developers. Massive pubsub system. Done tests with 1M
concurrent users on 200 small-node EC2 cluster with end-to-end latency mostly
< 1.5s.

Texspace - spatial search server based on Microsoft Research paper. Fuzzy
spatial search capable of working with completely unstructured queries. Had
pretty big issues with processing, indexing and analyzing big amounts of data
due to slowness of all DB solutions available at that time. Writing geometry
algorithms in Erlang is not trivial due to immutability, since all of them are
described in imperative languages.

IM subsystem for microblogging site (which is pretty big in Asia) -
implemented 5 proprietary protocols. Erlang is perfect for such tasks! The
only issue was lack of documentation for proprietary protocols ;)

Map routing server - implemented optimized A* search with route graph
generalization. Writing fast A* is tricky in immutable world. Books like
Okasaki's "Pure Functional Data Structures" FTW!

WMS Tiling / TMS service with caching. Simple and easy task for Erlang. Mnesia
+ file-level cache.

Overseer - Nitrogen + CouchDB app for tracking employees time by doing
screenshots via VNC and marking them as "work" / "not work". A breeze :)

Erlang FTW!

------
iamaleksey
SMS/MMS gateway (aggregator) with SMPP, CIMD and MM7 support and AMQP
interface (RabbitMQ as a broker). Also building a simple SMSC right now.

It's a perfect fit.

------
metabrew
I'm currently building a web-based IRC client with bouncer-like functionality
- ie, stays online when you close the browser, remembers backlog etc.

Built github.com/RJ/playdar-core in erlang, after initially writing it in
C++/boost

------
ams6110
Not mine, but there is a CMS built in Erlang: <http://zotonic.com>

------
rubyrescue
\- we wrote the backend of <http://bidit.eu> in Erlang (front end in Rails) \-
we're writing the backend for a TV co-viewing chat application for the iPad in
Erlang right now \- we're rolling out a new Nitrogen-powered website for Inaka
Networks this weekend

------
martinjlogan
I built one of the fist comprehensive call detail record collection systems
for a SIP voip network. I also built a least cost routing system for same.

A options feed aggregation and charting backend.

A package management system.

many many other small apps.

Cheers, Martin Logan erlangcamp.com

------
Groxx
Not mine, and though I can't find a page on their site that states it,
Vendetta Online[1] uses Erlang heavily for their server code. I went to a
conference they spoke at a while back, they've said it works wonderfully for
them and lets them handle large amounts of players easily.

They are also working to release a game engine (NAOS[2]) built from their
server code, which also uses Erlang pretty heavily

[1]: <http://www.vendetta-online.com/> [2]:
<http://www.guildsoftware.com/products.html>

------
nivertech
I did several small Erlang projects and now working on two relatively large.

Erlang is a good fit for:

* Irregular concurrency: Task-level, Fine-grained parallelism

* Network servers

* Distributed systems

* Middleware: Parallel databases, Message Queue servers

* Soft Realtime / Embedded applications

* Monitoring, control and testing tools

Not so good fit for:

* Concurrency more appropriate to synchronized parallel execution: Data Parallelism

* Floating-point intensive code (HPC)

* Text Processing / Unicode (Unicode support now much better)

* Traditional GUI (some improvement with wx now)

* Hard Realtime applications

* Extensive interop with other languages/VMs (some improvement here with NIFs and Erjang - Erlang on JVM)

* No Windows 64-bit VM

I using Erlang for the things listed under "not so fit", because it's
advantages out-weight some incomvinience.

------
brianjesse
I built a real-time news widget that streams (hyper)local news and public
records (marriage licenses, liquor permits, etc). Media companies partner with
us <http://nozzlmedia.com> we've doubled the time visitors spend on their Web
pages.

Erlang was a good fit for us, mostly because of the tools available around it,
such as Nitrogen <http://nitrogenproject.com> a powerful JavaScript/Erlang
framework. Erlang apps seem robust and lean.

------
zaphar
Started etap: <http://github.com/ngerakines/etap> which ngerakines took over.

Iterate: <http://github.com/zaphar/iterate> an agile user story tracking app.
Which is mostly useable but still in development. Used nitrogen which I found
to be a really good web framework for dynamic applications.

------
mononcqc
Mostly toy projects, many of them still in development.

A little homemade web framework. A blog compiler to generate a static site. A
one-on-one chat client to be integrated with other already existing websites,
a browser-based version of the Apples to Apples game, a brainfuck interpreter,
maze solver, etc.

------
jayair
We have an erlang cluster that runs all our services @ <http://thecadmus.com>

High number of processes distributed across multiple nodes with supervisor
trees taking care of them all has worked well for us.

------
bosky101
\- a key-value stored modeled caching system that can run functions during
set/get, have expiry, counters, counters that wait for N as a buffer before
running transactions, and machine learning - all as primitives. the tail-
recursive processes acting as mini-servers holding state and the best thing is
that you have a mini-automatic scaling if you combine this well with expiring
of processes. increase load when there's a need. they die when there's no more
need/lesser traffic.

\- a distributer crawler that works with python. from a typical rdbms approach
that brought down creating inverted index's from more than an hour to few
seconds. the euroka moment was when we stopped porting/writing for another
language, sequentially, for-loop style approach to a more - processes and
message passing. big 'why didnt i do this moment' a few years ago!

\- a website , dashboard for the ad/content network using yaws on the LYME
stack

\- and btw - all that rumour about unicode is just bs. actually it was a
blessing for us that content aka strings are just integers which when
converted to binaries not only takes less space, pattern matches like a dream
but all of a sudden your sytem can handle any language because to it - it's
all just integers, english or swahili. that was a huge bonus even when getting
investors, new untapped market opp's.

\- an ad network, ( content network i guess without saying)

\- a fire-and-froget thumbnail/screenshot system with queues & talking to
imagemagick . i love writing and fire-and-forget tools, and with erlang, its
so darn easy to write proceses's that have a life of their own. sure at some
point you need to compromise on erlang's unique ability to handle RAM vs
giving that up to research on how to handle large mnesia tables ( mnesia =
inbuild db that has a limit on size of tables) . but it's worth the tradeoff.

case in point - our idea of scaling was...'reducing nodes' ie increasing
efficiency

-2008 a million hovers in the year, 4 nodes

-2009 a million hovers every month, 3 nodes

-2010 a million hovers every week, 2 nodes i could bring it down to 1 node for doing the tasks as well, on just one 1 8gb RAM node.

the idea of reducing your costs, means that much lesser to profitability. (
we're now profitable, pretty much on auto-pilot on the backend, the 3-4 member
team concentrating on increasing toplines,sales now )

in addition... \- a wrapper to iui for nitrogen framework for building iphone
webapps easily \- wrappers to tokyocabinet ive contributed to called medici \-
remember writing wrappers to other things like rrd, aws utils, etc

all of the above have been used at hover.in, more at
<http://slideshare.net/bosky101>

there's nothing i can think of that i wont/cant do in erlang anymore.

~B @hoverin

~~~
lzw
Let me guess, all this over engineering nonsense set your startup back 3
years.

------
wagerlabs
OpenPoker

