We've been using Elixir almost exclusively for a couple years, and I can't speak highly enough about it. There are clearly problems where beam is ill-suited, but otherwise, I almost feel religious about it.
I've worked for many years with many other languages and runtimes, but beam, OTP and Elixir (and the synergy between them) are better tools. As a foundation, message passing and isolation are the only realistic weapons I've seen against most common causes of technical debt (coupling). It isn't a silver bullet, but it tries hard to help you. And, importantly, it's approachable.
Another way to look at it is that it delivers many of the same benefits of SOA/microservices, without the significant infrastructure/deploy/devops overhead.
Asynchronous messaging is overly complicated for request-bound tasks in web applications when using an external message queue. When it’s a first class feature of your language though, it’s way less painful than it would be with something like Python and SQS. If I were going to build something that was heavily built around asynchronous messaging, I’d definitely want to do it in Elixir. Otherwise, I tend to prefer REST or RPC except in situations where messaging is really necessary.
Seconded, though I don't mind Erlang as much as some. The most complex stuff I ever built was built long before I knew about Erlang (and before Erlang was available to outsiders afaik). Message switches and other distributed systems would have been impossible with the degree of reliability required without the kind of architecture that message passing enables.
I did it using QnX re-inventing quite a bit of the Erlang philosophy along the way, if I had had access to either Erlang or Elixer at that point in time I would have been overjoyed, that would have saved so much time.
Is Elixir being used extensively outside Silicon Valley? Can you list some well known success stories? Looking at job listings and open source projects, it doesn’t seem to be widely used. In fact, I don’t know anyone outside Silican Valley who has built anything non-trivial in Elixir.
I am not saying it is bad. But I’d like to know about it before I dive in.
Not publicly documented, but CNN's ingest scheduling system (basically the backend that says "at this time, record this incoming feed, on this encoder, associate it with this metadata", and all the ensuing complexity that comes with finding a free encoder, responding in the event of hardware failure, allowing users to plan recordings that there isn't space for, etc) is in Erlang. Because it needed to be bullet proof; if it goes down, CNN doesn't record incoming video.
Had Elixir been mature at the time, we would have used it instead.
That's interesting. Do you guys have to build a lot of custom tooling in TV? Are CNN's requirements particularly novel compared to other stations with live programming and remote feeds?
I've since left, but yes and no. Video at that scale requires some unique hardware, which in turn requires some unique software. But for broadcasters of that scale, many of the problems are the same.
We actually had, prior to this system, been relying on one built by a French company, that cost hundreds of thousands, possibly millions, per year in licensing. Updates were infrequent, often included regressions, but it mostly worked.
This started as a way to get some of the smaller bureaus to have some basic functionality, and grew into a system that was deployed to every bureau, eventually replacing what we had in Atlanta (that previously mentioned system). It was extremely popular, and the business stakeholders even characterized it as the biggest success they've had from our department.
One of our group's VPs showed it at an industry trade show, just offhandedly, and an exec from another broadcaster indicated regret of having just signed a deal to purchase the aforementioned system above, rather than attempting to roll their own (seeing the success CNN had).
It's not a key differentiator for CNN, but they also don't have software sales as a core competency. So they have basically best of breed software sitting there unnoticed. shrug C'est la vie.
It also has the benefit of, since it's Erlang, largely just working without issue. There's not many there still with Erlang knowledge, but that isn't really much issue for another 20 years if it continues just working without issue.
Cabify, an Uber competitor from Spain operating mostly in South/Central America and recently valued at 1.5B has been moving from a primarily Ruby stack to Elixir over the past year.
Why exclude the success stories in Silicon Valley - seems like an arbitrary decision to exclude Discord say or Pinterest?
It does seem there aren't that many very big companies outside of Silicon Valley using Elixir but maybe that's a factor of Silicon Vally having the vast majority of winners period.
Well, I don’t live in Silicon Valley, and as a result, the company I work for is not a Silicon Valley company. If I am going to convince the management to invest in Elixir, I need to be able to answer questions like this. Pretty much everyone I have spoken to about Elixir, inevitably brings up this point.
I don't understand why you make the distinction between Silicon Valley and not Silicon Valley because where something is done makes absolutely zero difference as to if it's a good idea to use software or not.
If there was a text editor that you thought was good but was only in use in Silicon Valley would you still recommend it? I've never ever in my life discounted a piece of software because it was only in use in Silicon Valley.
Maybe you could expand on the argument because I don't understand it clearly!
Can you share a bit about the deployment story. I haven't really been able to get a good view of the automation behind that. Granted, I have yet to develop anything actually deployable, but would love to hear some opinions about that.
My point is that you can achieve many of the same benefits of microservices within a single app (or "vm" in erlang terms).
So you get isolation (beam processes), decoupling (message passing) and resilience (supervisors) without having to deploy and maintain N different apps and the pain points that come with that (monitoring, discovery, integration testing, debugging, ...)
Our company has other systems, in other languages, and a pretty involved deployment process (docker+nomad) so our deploys are pretty specific to that. But, I can tell you that as a developer, having only a few "apps" that internally have hundreds of services, has more or less freed us up of having to worry about any of that cognitive (and performance) overhead.
And to further add to this. When he says "same benefits of microservices within a single app", that's because there's this concept of an "Umbrella app".
Basically you could split up your apps and hack on them independently (but actually have a good way to share dependencies if needed) and at deploy time you could choose to deploy all of them, or just the ones you want.
Phoenix (Elixir's goto web framework) also tries to nudge you in a good direction to split up your code base in a way that allows you to separate the specific pieces of your app. This is really good for people who just want to make maintainable projects. It's sort of like the first step in splitting up your application without dealing with any type of real additional complexity since changing things around just involves renaming and moving files.
I would say they are complementary. Kubernetes is a standard solution for packaging and deploying apps. If you run a lot of apps, you want your ops team to be able to manage them without knowing all the details. You get better utilization of your servers by allowing them to deploy standard units and manage resources at a higher level.
You can put an Erlang/Elixir app into a container and deploy it, but containerization doesn't provide as much value as it may for other languages. The Erlang VM doesn't depend much on the operating system. The mix build tool tightly manages runtime library dependencies, and building a release combines the VM and the app's libraries into a single tarball that you can deploy as a unit. So we don't find it particularly useful to use Docker/Kubernetes to deploy.
Erlang/Elixir makes it very easy to take advantage of all the CPU cores available, so you generally benefit from having one instance with lots of CPUs instead of lot of little instances. And it can be a lot cheaper to do this in a dedicated server environment instead of the cloud.
So on the whole, Kubernetes is not that big a win for Erlang, but they work fine together.
I also want to know this. We have a team of 3 people maintaining our microservices infra (services written in NodeJS), out of ~15 developers. It feels like a waste, these (monitoring, logging, deployment etc) are all problems that have been solved many times over.
I'll share our deployment stack with Elixir, which is pretty hands off. We use docker with AWS ECS and have a simple release process:
1. Our jenkins CI builds a docker image which contains an erlang release (using distillery).
2. This image gets pushed to ECR (AWS's image repository), every version is tagged with a $GIT_REF.
3. We have another jenkins job which updates our AWS ECS Tasks to use the new version of Docker images.
4. AWS ECS now spins up a few new containers and drains out the connections from the old ones.
We run in cloud or dedicated server instances. We deploy using Erlang "releases" using systemd for process supervision. We build on a continuous integration server and deploy using Ansible or AWS CodeDeploy.
I've worked for many years with many other languages and runtimes, but beam, OTP and Elixir (and the synergy between them) are better tools. As a foundation, message passing and isolation are the only realistic weapons I've seen against most common causes of technical debt (coupling). It isn't a silver bullet, but it tries hard to help you. And, importantly, it's approachable.
Another way to look at it is that it delivers many of the same benefits of SOA/microservices, without the significant infrastructure/deploy/devops overhead.