I’ve been doing Erlang professionally for the last few months. I have to say, it’s a breath of fresh air after decades of Java/Scala/
/golang/. It’s unfortunate it isn’t more popular.
OTP and Erlang make it so easy to build distributed systems end to end. It makes my past professional efforts look like a shoddy amateurish joke. If you find yourself needing to build a distributed stateful service, Erlang/OTP should be the top choice.
I'm struggling recently to build consensus with my team that Elixir is the right technology choice for a core element of our infrastructure. I've figured I just need to do it myself to demonstrate the value and then snowball from there.
I saw a funny comment on another thread, where the PM was unsure if Elixir was good/stable/whatever enough to build the core business service on.
The dev responded that 3 of their primary third party services were built on Elixir, so they're worried about using it in house they should probably drop those too.
If the success of Cisco, Whatsapp, Discord and nine nines worth of uptime isn't enough... you can link to our stodgy Fortune 100 company using it successfully!
Thank you! I worked quite a bit on the HL7 one! And I started on one for FHIR - but it's such a pain to work with I put it aside and considered releasing it as DumpsterFHIR or TrashFHIR. I don't think the company would be keen on that, though :)
what might explain this? I mean hypes and infatuations are common but tangible advantages usually win the day because in the end people will pick the right tool for the right job.
is it that distributed systems are not actually as big a domain so it does not get much press?
> tangible advantages usually win the day because in the end people will pick the right tool for the right job
No, they don't. They pick the familiar and the easy-to-hire-for. I've worked on systems (written in C++, Java, others) that would have been perfect for Erlang, but at most I was able to present a prototype. No one is interested in a better solution that they can't hire for or have to retrain their staff for. (The familiar and the easy-to-hire-for may, by some metric, be the right tool for the job, but it isn't necessarily the right tool by technical metrics.)
I am pretty strong with Elixir, and I've done a little bit of Erlang. Anyone who understand the basics of Erlang features that get used in Elixir should be able to program in Erlang with a bit of a ramp up period.
Not too long ago I interviewed with a company that only uses Erlang, not Elixir, and they rejected me because I am not an Erlang expert. There wasn't anything I could say that would change their minds. I even volunteered to do a coding challenge of some type & they still wouldn't consider me further.
When I worked at an Erlang shop, the director’s hiring strategy was to hire generalists willing to learn Erlang. He also hired remote, years before the lockdowns. He built distributed teams that builds distributed systems.
> Erlang, not Elixir, and they rejected me because I am not an Erlang expert
Wow... was this an actual hiring manager or someone, or like an HR person with a checklist?
I've interviewed for one or two Elixir positions where the opposite thing happened: I talked with someone who had a checklist. I haven't done Elixir professionally even though I have extensive Erlang experience and that was that.
There is a pretty stark difference between "has architected an application with OTP components" and "can write linear code in Erlang or Elixir", but even then you'd probably be able to fit someone in unless you're hiring for that kind of senior/architect position.
It was a team decision. I interviewed with the hiring manager and three developers. It sounded like I was the first Elixir person they'd spoken to, and just decided it was outside of their comfort zone.
That's what I did! I learned Elixir in my spare time and then started a job that used Erlang. There's some quirks that I had to get over, but it wasn't a huge change over.
> > [...] people will pick the right tool for the right job
> No, they don't.
I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical).
I assert the following:
Scenario A: I have a literal mathematical proof that erlang is the best choice. I implement the solution, but due to the market changing (or an earthquake or ninjas literally destroy the server room) the overall project fails.
* Me: But, I had math proving everything out. Ultimate Boss: I don't believe in math, you're fired.
* Boss: Well, it failed, but look at this incomprehensible math that I don't understand. Ultimate Boss: You've obviously been hoodwinked, you're fired.
Scenario B: I have a really good story that includes phrases like: "best practices" and "industry standard", and is doing exactly the same thing as everyone else. The project fails because the tool isn't great.
* Me: It failed, but here's a good story. Ultimate Boss: Well, it looks exactly like everything else, so I'll look for a different nail that's standing out from the crowd to blame for this so that the muggles don't get nervous about taking non-chances in the future.
* Boss: Here's this great story that I totally understand and can embellish upon because it operates on storytelling and not category theory. Ultimate Boss: That's a really good story. I'll have to steal it myself to give to the board to handle some unrelated problems of my own. Better luck next time.
Even the scenario of master craftsman using erlang because it's the right tool for the job is more about a story about an uber expert who is off doing wizard stuff than it is about an actual merited set of reasoning.
> I'm in strong agreement. People pick the option that has a good story associated with it. Having a good story protects you in the event of a failure. And it protects your non-technical boss (or your boss' non-technical boss if your boss is technical).
The "nobody ever got fired for buying IBM" defense. Which has since been updated to Oracle, Microsoft, Java, C# (for a large portion of the enterprise world), Python, and Javascript.
Best-in-class that isn't mainstream is a hard sell to employers if they are strongly risk averse (which many people, if not most, are).
Sounds like they're the prototype for a new company. WhatsApp ran rings around competitors with a small team using Erlang; can you do a better job than the company that rejected it?
Too low a margin and too few customers to make it worth it for me to try and jump into those niches. Honestly, many of the benefits of using Erlang for those systems would have been similar to any other more service-oriented or decomposition of monoliths. They had complex systems doing too much, but that would have benefited from the really good binary processing portion of Erlang and it's relatively easy horizontal scaling story. But when the number of customer (organizations) can be counted on one hand, and the number of users is less than 1000, and there's already competition you know it's going to be hard to break in. They'd already hit rock bottom pricing on their contracts.
I work for a smallish supermarket chain (I assume, by the US standards) - we have about a 1000 stores in 5 countries, about 40-45K tills. At peak trading my teams' services handle a combined 250K tps. We run a bunch of distributed systems in the cloud, across two DCs and some other weird legacy locations (there are even a few mainframes) - the business is really old and went digital before some of my colleagues were born :) (Anyone remember teletext? Yep, we were on that too).
So, Elixir/Erlang combo seems like a perfect fit. Instead, we are having all remaining Elixir and Erlang systems ripped out and re-written in Java.
The reason is simple - readability and hire-ability. The cherries on the cake - Java's "good enough" for what we do, the tooling, the quality and availability of reference resources and subject matter experts - are just that, cherries. But the ability to hire and onboard new engineers is THE thing - it took us 3 years to fully hire out the headcount we were allocated for just our little team and that is in Java. Hiring for Elixir and Erlang has been such a pain that we are having to pay contractors while we re-write the current Elixir and Erlang systems into Java. Our domain is somewhat complicated - most people who have not worked in food retail generally would not imagine it to be so - so onboarding people to the domain knowledge takes 3-6 months and that is without "This is what? Erlang?" barrier. That is where readability of Java and non-FP vs Elixir/Erlang and FP comes in as well.
I guess what I am trying to say is that, yes, the right tool for the job and all that but good enough tool with a significantly higher hiring pool is proving so much better.
It would be interesting to see where things are at in a few years. I get the problem. But not sure the right decision was made. I've seen the "leave esoteric/boutique/niche language for more commodity platforms" play out a number of times over the years. The results are very mixed, and hard to infer because of the many variables. But I don't usually see the "we're so much better now" after the original zeal for the exodus pans out. Usually it ends up being more of a "you had a problem so you did X. Now you have two problems" in my experience. I wish you and yours the best of luck.
Last year, according to Stackoverflow polls, Elixir was the second most loved language, just under Rust. There is a reason why people who knows it, loves it.
Elixir’s runtime, unlike many others, uses preemptive lightweight processes. Unlike async-only systems, a runaway process won’t block out other processes. That’s just the start of the kind of advantages in writing with Elixir/Erlang/OTP
It’s been battle tested for years. I think many of the hype cycles for distributed systems reexplore or reinvent what Erlang and Elixir has to offer.
Distributed systems are everywhere in big tech. So that's not really the issue.
A lot of it is unfamiliarity with the tool and unfamiliarity with the paradigm. And of course, you can do distributed systems with other tools that people tend to be more familiar with, so go with what you know. Erlang is adaptable, but was built for a Telcom setting with a lot more permanence than today's popular cloud model with nodes coming and going.
On the other hand, an awful lot of large scale chat systems start as ejabberd and adjust from there; sometimes rewritten in another language, sometimes rebuilt in company specific Erlang while the service is running, sometimes without muvh adjustment.
Elixir/Erlang/OTP actually holds up well even with modern cloud environments, including Kubernetes.
For example, you can effectively architect with microservices without actually creating more deployable artifacts. This is because each GenServer is already a microservice. You build it from the ground up reasoning through failure modes and coordination. Those are easier to separate out into its own deployable artifact when the time comes, if it ever does.
People hear about code hot reloainding (where you deploy new code without shutting down the production environment) and think that’s the killer feature of the BEAM vm. Code hot reloading is a narrow use—case of a more general capability — you can directly write or patch new code in a running system in a live environment through the REPL. That has been very useful for improving the system, to stop the bleeding or identifying bottlenecks and trying out fixes, even when deployed as ephemeral pods in Kubernetes.
I find C/C++, et al curly brace syntax and semi colon at end of line, maybe; syntax absolutely abhorrent. Erlang reads like a sentence commas, and period at end. There's very little syntax and it makes sense. The problem it is different than what 90% people learned programming on (curly brace languages).
But the hard part of Erlang is OTP / understanding the concepts or rather unlearning the concepts from other programming paradigms.
i spent a couple of memorable weeks a few decades ago between jobs reading up on erlang, including joe armstrong's (RIP) wonderful phd thesis. when i started, erlang looked "strange" (i hadn't done much prolog in uni). but after being immersed in erlang for a few weeks, when i returned to look at some C/C++ code, they looked positively hideous compared to erlang's higher level functional, and prolog inspired pattern matching syntax.
It does look pretty weird if you have only ever worked in an imperative language.
If one can get past the initial impression though, they'll find it's actually a very simple, nice language. Much simpler than Scala/C++ and probably even Java these days. Also simpler than other functional languages like Haskell. I picked it up in about 1-2 weeks. There's very few gotchas in the language or hidden complexity. It flows well.
I'll also add the language suits the usage pattern. Pattern matching works very well with the gen_server approach. And the binary syntax is so nice if you need to deal with over the wire protocols, as is common in distributed systems.
You're preaching to the choir. I've been singing the praises of Erlang for about 15 years now - since I discovered it via a competitor's open source project.
I like it. It took a bit of getting used to ("bit" here used to mean "pretty minimal amount"). I didn't find it to be any sort of hurdle, and Erlang meshes quite well with how I think about programming. It's my favorite language of any that I've used.
I completely agree -- it blows my mind that the BEAM/OTP is not more widely used. It is such a phenomenally well-designed system and it suits modern use cases perfectly.
It's incredible for IO. It's not amazing at tight loop CPU intenstive tasks, but elixir and erlang do make it really easy to take something sequential and spread it across all cores (or across a cluster if you're so inclined).
It does have good FFI, like rustler for the elixir is great. It makes sure rust panics unroll into elixir exceptions and that takes some of the scariness of writing them.
That plus dirty schedulers means you don't have to be a good citizen and can write cpu intensive code in a faster language.
a platform that would make distributed computing less of challenge would be well placed today with all the data science / ML / DL / AI focus but its not clear if erlang has ever had any ambitions in this space.
the main reference that shows up if you search for erlang + hpc is from 2008
I beleive there is currently a PhD project in collaboration with Jose Valim (Elixir's creator) which is looking at distributed machine learning.
There is also the NX project (numerical elixir) which compiles tensors to GPU code making it much faster for those particular usecases. https://github.com/elixir-nx
This has things like explorer which is elixir bindings to polars, livebook which is like jupyter notebooks (with enforced execution order thang god), aswell as some machine learning libraries in axon and scholar.
There has also been some work with libraries like flow and broadway which allow pipelines to be constructed with the usual syntaxt `step1 |> step2 |> step3` but to be executed across cores (or cluster).
I can imagine orgs using a combination of the above tools to do some interested distrubuted data pipelining and ML.
I've been using Erlang for nearly 20 years, on and off, and I think a reason it's not more popular is that it has always felt like a tool that is really good for some things, but maybe not so good for this or that random task. If you pick Python or Ruby or Java, you know you can probably do the thing in question, whatever it is, even if that language isn't the perfect fit for it. As well as the next thing and the thing after that. There's something in your toolbox for it. With Erlang, the toolbox didn't always have that depth. It's gotten better with Elixir on the scene, but there were times when I would wade in and fix upstream bits and pieces with Erlang libs. That very rarely happens to me with Ruby.
That said, it is a fantastic system for some problems and I really miss working with it.
Scripting, string munging, any sort of small task, anything which runs locally (unless it’s changed significantly erlang’s distribution story is the worst).
One day I will switch to Erlang/OTP ,I really want but it's not easy.
We need a larger ecosystem, because most of the times when people like me look for what language to use to solve a problem, we try to select the best / easiest tool for the job, and many times if the ecosystem has easy to use tool, we go with that.
Not all developers build distributed systems and even tho I'd love to, many times I don't have the capacity to build things I need from scratch so that leads me more towards java/js/python sadly.
If only I could convince people of learning and giving it a try in professional context, then I would already have worked with it. But few people are willing to learn.
Have you used Elixir? I ask because the Elixir books mention that setting up GenServers is easier in Elixir, with behaviours and callbacks you don't have to write code for all the functions in a GenServer if you're not using them, while it's required in Erlang.
I am not at all saying this is you, to be clear, but one problem I have had in the past looking for Erlang jobs is that occasionally you find people who view it as some kind of magic sauce, or "I'm curious to see what it's like".
In the former case, they think just writing their code in Erlang is going to give it magic scaling powers. This is probably less common these days, but I recall a while back when Erlang first got a bit of hype, there were people who weren't thinking very clearly about it.
In the latter case, sometimes it's not the best tool for the job but someone wanted to learn the intriguing new thing.
Edit: and of course there are great use cases for it. See my other comment elsewhere on a system where it was a great fit. I'm always curious to see where others are using it successfully.
Plenty of people build highly tolerant, robust distributed system of top of regular languages when Erlang exists, and no one bats an eye. Erlang is no magic sauce, but for its niche it is unrivalled and yet routinely ignored in favour of subpar platforms because of dubious concerns such as "there are no Erlang/Elixir developers around", which is nonsense.
The true reason is that there are plenty of engineers and architects that have no exposure to the BEAM, so have no idea how productive they could be developing their distributed system on it. They simply do not know what Erlang & co. could do for them.
Certainly, but for each and every small task you do, you don't evaluate a new language for it. If you have a web application that's done in Rails and have a bunch of Ruby programmers, if you need to, say, add a simple web scraping system to it in house, you're likely going to reach for a Ruby tool at first.
On one side, you have shops that are like "we only use Java". They're maybe missing out on some good tools for various tasks. The other extreme would be a big list of languages, and that's difficult to maintain as people come and go.
Erlang is not just a language, is a different paradigm, and it is worth evaluating, just like deciding whether you need a native language, or Python's enough.
Are you building a distributed system? Are you building a highly concurrent IO-bound server? Are you building a system that needs to be fault-tolerant? Going for anything other than Erlang/Elixir is a misinformed choice.
For me the question often hasn't been "is it distributed" but "how often do you need to spawn some async task (that can be run supervised, or unsupervised, in a fault tolerant manner)".
I find that is often a lot more relatable to average-joe-tasks than bigger picture distributed ideas, even if they're actually the same thing at their heart.
I just don't want one request to blow out the system, or some sporadically turned remote API to hose everything, Elixir gives me that (with the simplicity other languages often need other stuff stuck onto to handle).
If you have something like a web site with a database, where 1) the database is critical to the normal functioning of the system, but 2) you still want the site up to let users know that "it's broken and we're fixing it", and maybe also send some alerts out, you might want something more like an exponential backoff or some other more involved strategy, which isn't shipped with Erlang.
There are infinite ways of dealing with failure. Erlang gives you the building blocks, if you need something more complex, you're supposed to have to write some code. How is this surprising?
In fact, I am writing this week a dynamic process pool in Elixir which is so peculiar in its requirements I don't think I've ever seen anywhere, and it's in total 500 lines of code spread over 2 GenServers (+ Registry and DynamicSupervisor). I reckon it would easily be 25x as much in C++ or Rust.
If you want exponential backoff, I suppose it should be pretty easy to implement with a custom Supervisor (though the best practice would be to have a sidecar GenServer to deal with the backoff/alerting business rules, and using a regular Supervisor.)
This is the kind of thing I'm talking about though... we go from "Erlang is a piece of cake for distributed systems" to some pretty advanced concepts and telling people to "roll your own" for an extremely common use case: not wanting your entire application to be taken down by an unreachable external service.
My company has a real time communication system: Real time video/audio/chat. Elixir/Erlang works fantastic for this, as you are doing a lot of real time message passing, creating temporary sessions/channels for communication, and monitoring tasks.
We've been able to easily scale with minimal effort and have never had downtime.
i've always wondered : it seems to me that handling hordes of process is today made quite simple thanks to tools like kubernetes, and middlewares like queues or databases of various kinds for process 2 process communication with various guarantees.
What's not solved by erlang is IMHO the hard part : how does your system ensures data persistence, availability and consistency ?
Rolling my own cassandra or kafka or distributed db isn't really something that enchants me, and i don't see how the beam can makes this any easier ( they are fundamental problems, not technological ones).
The thing that Erlang and Elixir have different is the BEAM VM and its support of concurrent processes at the VM level. So there are some things that they can do that Akka can't. For example, the scheduler is preemptive and keeps processes alive even though there may be rogue processes pegging the CPU.
TLDR: GC and runtime behaviour generally are built from the ground up in OTP with this kind of concurrency and programming model in mind. Its advantages can be approximated but not fully replicated in "general purpose" runtimes like the JVM.
Consider that the JVM model is for long-running shared memory systems. So its GC in particular is built around that model. Erlang/OTP as I understand is a lot of short lived processes that communicate through messages, sharing little state. Designing a GC for a system like that has entirely different trade-offs. Isolated short lived actors can throw-away their entire memory state at process end, without waiting for a generational collector to discard the garbage, etc.
And that's just the GC. In general in a system bolting "actors" onto another shared memory language it's always going to be possible to "escape" that framework and do things in another paradigm. That might be fine for the real world, but it negates some of the advantages that OTP brings to the table.
Yeah kinda. Akka is an excellent option if you’re in the JVM ecosystem. I find it’s encumbered by a lot of complexity and cost though - which kinda comes with Scala and the JVM. It’s also a kitchen sink that tries to be everything for everyone - Akka was positioned to be the Spring of Scala, after all.
Whereas Erlang/OTP is a very pragmatic, simple language and runtime geared solely towards building distributed services. The gen_* behaviours are awesome and I’ve never seen anything like it (at least in terms of simplicity and practicality) in other languages or runtimes.
Kubernetes and the BEAM are actually a great match. Kubernetes for management and supervision of nodes/containers and the BEAM for management and supervision of processes. I see them as complimentary technologies.
I began my journey into functional programming a couple months back with Erlang. Interestingly this language helped me truly comprehend the functional like ways of Rust.
My only gripes would be that I've had to switch to Elixir as it's more developed for web development. I actually like the syntax of Erlang more (I don't come from Ruby) and it's a shame that they're split. Also, in Elixir, I attempted to write the fibonacci sequence using memoization and the solutions I saw online were horrendously ugly. Maybe it's because I'm still a beginner but I did see an example in Haskell and it looked so much cleaner.
I love how simple the actor model is and how it just works in Erlang. No ifs, ands, or buts.
The feature hiding in Elixir, and I assume Erlang, is the binary pattern matching, on either whole bytes, big or little endian, down to individual bits in a byte.
I'm helping with a project to recreate the old Prodigy network, and the combination of FP principles and the ability to slice and dice bytes like I did with C in the '80s has me hooked on Elixir.
OTP and Erlang make it so easy to build distributed systems end to end. It makes my past professional efforts look like a shoddy amateurish joke. If you find yourself needing to build a distributed stateful service, Erlang/OTP should be the top choice.