Erlang gets a lot of stuff right for scalable web based stuff, and even though there are many of its influences that have by now made it into other languages and eco systems it is still amazing to me that such a well thought out system is run with such incredible modesty. You'll never see the people behind Erlang be confrontational or evangelists, they just do what they're good at and it is up to you whether you adopt it or not. And that is an interesting thing: the people that are good at this are writing code, not evangelizing. If I had to reboot my career I'd pick this eco system over anything else, it has incredible staying power, handles backwards compatibility issues with grace and has a community that you can be proud of joining. Modest, competent, and with a complete lack of drama.
> You'll never see the people behind Erlang be confrontational or evangelists, they just do what they're good at and it is up to you whether you adopt it or not.
The big open source projects where pretty much all like that in the past, in the 80's/90's/early 2000's - in that respect they feel like a pleasant anachronism before everything needed to be promoted/self-promotional influencer like, the users did the evangelism but the creators where usually much more chill.
Obviously the vast majority of open source projects are still like that but there is definitely a lot more in your face promotion of things that feels different somehow almost aggressive/corporate style even when there is no paid product.
Not knocking the ones who do it, if it's open source they can sing it from a mountain top for all I care, the license it's under matters more.
I think what has changed mainly is that today we have tools, languages and entire ecosystems that exist only as means to support someone’s product line.
Take Swift for example. A giant gatekeeper of a corp decided to
make it the only (reasonable) way to build apps and so it exists, powered by countless indie developers constantly creating content around it. Would Swift be a thing without everyone being forced to use it? I don’t know, but I don’t think so.
So in some ways we’ve traded unique and effective solutions to “popular and mainstream” things that scream the loudest. You wouldn’t get fired for choosing Swift. Or Azure.
Yeah, Swift was born because Apple had an ecosystem and they needed to fill it with a language. Erlang was born because Ericsson had a problem and they needed to solve it with a language.
When I was working with it (I was there, 4000 years ago) there was some talk about Swift for the server, but neither obj-C nor Swift ever really breached containment of the Apple ecosystem and -tooling. Which is a shame because at the time I enjoyed working in XCode. Who knew using a mouse swipe to go back in your code would be so natural? Not any other IDE developer, ever.
Last time I worked with it it felt very sluggish and buggy though, in theory building UI elements with SwiftUI is great, in practice it was slow and needed to restart very often, and that was with simple components.
That is why I don’t like those ecosystems. They’re all relying on magic (code generation and indexing) for everything instead of just providing a good notation.
If you’re creating that closed of an ecosystem, at least learn from history and create something like smalltalk.
> The big open source projects where pretty much all like that in the past, in the 80's/90's/early 2000's - in that respect they feel like a pleasant anachronism before everything needed to be promoted/self-promotional influencer like, the users did the evangelism but the creators where usually much more chill.
I must have been living in a different world then. I mean maybe in the 80's and 90's but I feel like people acting weirdly obsessive about a piece of tech and going about evangelizing it every where, usually in a detached from reality kind of way, goes back to at least newsgroup, when suddenly you could have an audience outside of physical event (with their limitation and all). I mean there was the text editor flame wars, and I am sure you can find post like "why are you not using language/database/tool X instead of Y???!!" in the most ancient of mailing list and forums.
>somehow almost aggressive/corporate style even when there is no paid product.
For those who collaborate with open source for political/ideological reasons (which does not need be the case), it makes sense to join the battle for attention.
As long as the product isn’t compromised in the way, I think it’s very good to see open source influencers.
GNU Guix has a good blog, but I don't feel like they are very "marketing" focused.
It's hard to describe precisely, but a lot of free software projects do a good job of putting themselves out there in an unfussy way. There really is something refreshing and cozy about that.
i think this is especially true of projects run by people with deep experience in the field and are in it “for the love of the game”, and don’t feel the need to stunt on everyone in hopes that they are taken seriously.
I think it's because we are wired that attention is it's own currency nowadays. And it's also true. Even if there's no paid product, you get strength in numbers. If you depend on an open source library, it's usually better for you if others depend on it too.
Certainly an element of that but there are also cases where the superior product "lost" to the inferior product because the inferior one was better marketed.
So doing some level of promotion becomes necessary if you want users even when you have the better product - the superior product speaks for itself doesn't often apply any more.
And programming languages are in the lower end of quality actually impacting decisions. People are incredibly resistant to changes there, and just can't evaluate competing options at the same time.
What were the major projects that changed that? Rails and Prototype.js feel like early examples where there was a major cult of personality around their creators, but I don't know if that was really unique to them or something that was just an extension of the webdev / css / web standards blogger culture of the era— A List Apart and all the people who would got to SXSW every year to chit chat with each other on panels and share the hottest new way to make a row of tabs without using a <table>.
I don't think it's cycles, more like newcomers rediscovering the future.
I've learned Elixir in 2016 after a lull in my interest in programming languages, and 9 years later it's still my favourite environment by a country mile. It's not the language per se, but the BEAM, the actor model, the immutability — just makes sense, and doing things the C/Rust/Javascript/Python way is like building bridges out of cardboard.
For example, I've stepped into the world of game dev and Godot, which is fantastic and uses a regular object-oriented model. After trying to build a non-trivial desktop app with it, my thoughts are consumed by the fact that mutable state and object orientation is the silliest idea, and I'm speaking as someone that really got into Smalltalk and message-passing objects.
I don't even need actors and OTP, I just want some immutable data structures and functions operating on them. Erlang/Elixir are fantastic to build servers, but there is a sore lack of something closer to the metal within 80% the speed of a native language. I would build an entire operating system out of it. Why has no one put microkernels and Erlang into a blender? I know there's QNX, but it's still UNIX, not Erlang.
I have nothing but admiration for Erlang, and it is, without a doubt, one of the most inspired languages I've encountered in my career. But when I was at university in the late-ish nineties, they taught us Haskell as "the language of the future." So I guess some languages are forever languages of the future, but they still inspire ideas that shape the actual future. For example, Erlang monitors were one inspiration for our design of Java's structured concurrency construct [1].
If you're interested in another "language of the future" that bears some superficial resemblance to Erlang, I'd invite you to take a look at Esterel (https://en.wikipedia.org/wiki/Esterel), another language we were taught at university.
> Why has no one put microkernels and Erlang into a blender? I know there's QNX, but it's still UNIX, not Erlang.
That's a very good question. There are some even lesser known dialects out there that do this but you are going to find it hard to get to the same level of feature completeness that Erlang offers out of the box.
QNX and Erlang embody quite a few of the same principles, but QNX really tried hard to do this at the OS process level in a way that destroyed a lot of the advantages that doing the same under Erlang would have. I think the main obstacle is the fact that the CPU does not support reductions natively. Maybe you could take it a step further and design an FPGA CPU that implements the core features of Erlang at the hardware level?
That would be an absolutely awesome project. Usually when you can think of it someone has already done it so a bit of googling would be a good way to start with that.
You nerd sniped me a little and I'll admit I'm not 100% sure what a reduction is but I've understood it to be a measurement of work for scheduling purposes.
Oh that's a really neat find. I'm not sure how 'instructions' map to 'reductions' in the sense that if you stop when a reduction is completed the system is in a fairly well defined state so you can switch context quickly, but when you stop in mid reduction you may have to save a lot more state. The neat thing about the BEAM is that it is effectively a perfect match for Erlang and any tricks like that will almost certainly come with some kind of price tag attached. An interrupt is super expensive compared to a BEAM context switch to another thread of execution, you don't see the kernel at all, it is the perfect balance between cooperative and preemptive multitasking. You can pretend it is the second but under the hood it is the first, the end result is lightning fast context switches.
But: great find, I wasn't aware of this at all and it is definitely an intriguing possibility.
> That would be an absolutely awesome project. Usually when you can think of it someone has already done it so a bit of googling
I've done a bit of googling and research, nothing viable has surfaces, and I still haven't found the time to create a prototype myself, just some doodling around here and there. I do agree that it's an awesome idea, and it's been stewing in my head for a couple years now!
There are a lot of moving parts (scheduler design, probably needs its own high level language and bytecode, mapping capabilities onto actor semantics, etc.) that are outside what current OS-research seems to be focused on.
EDIT: I've just seen the mention of your QNX-like OS in your profile. Lovely! Any reason you haven't posted it on Github/Codeberg?
I've got a hobby OS you may want to check out. Crazierl is a just enough kernel that can run the FreeBSD build of BEAM as a single OS process.
Features include: x86-32 only, bios boot only, SMP capable, drivers in Erlang (there's nifs for managing memory with devices or i/o; and the kernel manages the interrupt controller(s) and has console output before userspace takes over), a kind of working IPv4 stack, dist!
It doesn't run on all my machines, but it does on some (it does some VGA things that I guess aren't well supported and also non uefi is iffy these days too. I typically run it in qemu and v86, but I think it will work in VMWare as well.
If it doesn't take much time, it's worth trying to get it to run in v86; it's so much easier to send people a link to a web page to poke at your OS than to send them an image and tell them to run qemu against it.
Edit: I misinterpretted --- you'll carve your OS up, not mine, that makes more sense!
Old comment: Feel free to carve away, just be aware that just because it's committed doesn't mean it works... I wouldn't take my memory management code, for example. There's some fiddly issues I haven't tracked down because it doesn't break consistently.
I got distracted (story of my life) by something very interesting that takes precedence but I'll return to it at some point, I've made up my mind about that.
v86 has been posted a few times [1], I'm pretty sure that's how I became aware of it to use as a target platform. It's a nice project, and they've taken three PRs from me (so far) that make it easier for me to run my weird OS :) Lots of other contributions here and there from others which is great, because I thought I was going to have to build a virtio-net device in v86 or write a ne2k driver in crazierl, and other people built and debugged the virtio-net so I can just use it.
> Any reason you haven't posted it on Github/Codeberg?
Yes, I don't want Microsoft to be able to pretend claim I gave it to them for some particular purpose which I didn't. They'll have to come and take it.
Feel free to use that code and do anything you want with it, and if there are no more seeds for the code let me know and I'll serve up the torrent.
It just got ingrained into pretty much every mainstream language, and most [1] of the wins can be had even when it's applied to certain parts of the code base only. Like, Java has immutable data classes (records), ADTs with pattern matching, etc.
As much as I like Clojure, I just don't think Lisps can ever become too mainstream, their readability simply repels too many people (and even though I am okay with reading it, I do think it's less readable than an equivalent Java-style code would be).
[1]: I would even argue that a properly mixed pure and (locally) side-effecting code is the happy ground, as they strengthen each other
FP language is very hard on novice programmers. You can write thousands of lines of bad javascript/java/python code, but you won’t write ten in FP without the whole thing blowing up.
And then there’s the whole evaluation instead of instructions. With FP, you’re always thinking recursively. With imperative, you can coast on a line by line understanding.
> JOSE: Yeah, so what happened is that it was the old concurrency story in which the Clojure audience is going to be really, really familiar. I’ve learned a lot also from Clojure because, at the time I was thinking about Elixir, Clojure was already around. I like to say it’s one of the top three influences in Elixir, but anyway it tells this whole story about concurrency, right?
I work with elixir daily and I would concur. elixir's semantics line up nearly 1:1 with the clojure code I used to write a few years ago. Its basically if you replaced the lisp brackets with ruby like syntax. The end result is a language that is much easier to read and write on the daily with the disadvantage of making macros more difficult. I would argue that it should be difficult since you should avoid using it until absolutely necessary. Lisps on the other hand, practically beg you to use macros as the entire language is optimized for their use.
It’s perennially in my list of languages to check out. It felt harder when I looked into it 15 years ago. Now that functional programming is second-nature, it should be much easier.
> Lets count the number of comercial sucesful released games that don't use mutability or object orientation....
1) I suspect game engines that Carmack designed towards the end of his career are built in a largely-functional style. He's on record [0] as thinking that writing code in a functional style is generally a good thing to do.
2) Running on the BEAM doesn't mean that you give up mutability. In my experience with Erlang, functions that return updated state are very, very, very common. Similarly, functions that mutate state external to the function [1] are also common... it's rare that a program that has no visible effect on the outside world is useful.
It's generally quite a lot easier to understand what your program does when most or nearly all of its functions take input and return output without mutating external state as a side effect.
[0] There's some Twitter thread of his that I CBA to find saying -in effect- "Writing in a functional style makes your code much easier to understand. Obviously, if you're writing a video game, you have to pass around pointers for efficiency reasons, but that doesn't preclude writing most of your code in a functional style."
[1] Such as in an ETS table, a external database, with a network call to a remote system, or even writing data to disk.
Oddly enough, you can write in a functional style in most any programming language. It's true!
And (as a fun fact) did you know that Crash Bandicoot, Jax and Daxter, and many other Naughty Dog games were written in Lisp? I expect that more video games sold at retail back in the day were -whether entirely or just in part- written with weird languages and runtimes than we would expect.
But, to answer your question: I don't pay much attention to who's doing what in video games, so I don't know for sure. Folks often talk about using Erlang for video game servers, but not so often about video game frontends.
I do know that Erlang's said to be a pretty bad fit for -say- 3D video games that require high raw performance. Sections 1.3 and 1.4 of the Erlang FAQ [0][1] provide a decent idea of the sort of things for which it is and is not a good fit. Particularly relevant would be these excerpts:
What sort of applications is Erlang particularly suitable for?
Distributed, reliable, soft real-time concurrent systems.
...
What sort of problems is Erlang not particularly suitable for?
...
The most common class of 'less suitable' problems is characterised by performance being a prime requirement *and* constant-factors having a large effect on performance. Typical examples are image processing, signal processing, sorting large volumes of data and low-level protocol termination.
...
Most (all?) large systems developed using Erlang make heavy use of C for low-level code, leaving Erlang to manage the parts which tend to be complex in other languages, like controlling systems spread across several machines and implementing complex protocol logic.
But you should really read those sections of the FAQ for yourself (while remembering that they were written like twenty years ago).
Also relevant is this excerpt from Learn You Some Erlang's introduction chapter in the "Don't drink too much Kool-Aid" section, written in ~2010: [2]
Erlang is no silver bullet and will be particularly bad at things like image and signal processing, operating system device drivers, etc. and will shine at things like large software for server use (i.e.: queues, map-reduce), doing some lifting coupled with other languages, higher-level protocol implementation, etc. Areas in the middle will depend on you. You should not necessarily wall yourself in server software with Erlang: there have been cases of people doing unexpected and surprising things. One example is IANO, a robot created by the UNICT team, which uses Erlang for its artificial intelligence and won the silver medal at the 2009 eurobot competition. Another example is Wings 3D, an open source 3D modeler (but not a renderer) written in Erlang and thus cross-platform.
Having said that, I play a lot of video games. Based on what I've seen, I expect that most indie video games these days could easily eat the constant factor introduced by using Erlang for coordination.
Having said that, with the existence of Unity and Unreal Engine, along with the documentation and assets built for both, why would you?
[1] Those FAQ sections have remained largely unchanged for something like twenty years. Computers have gotten much faster (and the Erlang VM and compiler have gotten much better) over that time, making the "constant factors" introduced by Erlang smaller than they once were.
> The example you gave about how functional laguages can make comercial videogames...
No. As I said:
> I expect that more video games sold at retail back in the day were -whether entirely or just in part- written with weird languages and runtimes than we would expect.
It's funny that you mention this, and it made me take some time to appreciate I've been working with Elixir full-time for almost 10 years now, and the entire experience has been so... stable.
There's been little drama, the language is relatively stable, the community has always been there when you need them but aren't too pushy and flashy. It all feels mature and – in the best possible way – boring, and that is awesome.
For me it took a tremendous amount of work to somewhat understand the OTP stuff though.
Its one of those languages where I can never be confident about my implementations, and thankfully it has features to check whether you have stale processes or whatever.
A language I am humbled by whenever I use it.
I love saying this but OTP is a really roughneck standard library. They just added shit to it as they needed it without apparently putting too much consideration into the organization, naming, or conventions.
It makes it very powerful but very disorienting and experience gained with one part of it often does not really prepare you for other parts. Usually each specific tool was created by someone who used it immediately, so it's all reliable in its way. But there is a lot of redundancy and odd gaps.
Elixir's almost extreme attention to naming, organization, and consistent convention is almost as far as you can get from this approach too. It's fun to have them in the same ecosystem and see that there are actually pros and cons to each approach.
Here's a trick to confidence in a BEAM system. If you get good at hot loading, you significantly reduce the cost of deployment, and you don't need as much pre-push confidence. You can do things like "I think this works, and if it crashes, I'll revert or fix forward right away" that just aren't a good fit for a more common deployment pattern where you build the software, then build a container, then start new instances, then move traffic, etc.
Of course, there are some changes that you need confidence in before you push, but for lots of things, a bit crashy as an intermediate step is acceptable.
As for understanding the OTP stuff, I think you have to be willing to look at their code. Most of it fits into the 'as simple as possible' mold, although there's some places where the use case is complex and it shows in the code, or performance needs trumped simplicity.
There's also a lot of implicitness for interaction between processes. That takes a bit of getting used to, but I try to just mentally model each process in isolation: what does it do when it receives a message, does that make sense, does it need to change; and not worry about the sender at that time. Typically, when every process is individually correct, the whole system is correct; of course, if that always worked, distributed systems would be very boring and they're not.
Erlang's hot reload is a two-edged blade. (Yes yes, everything is a tradeoff but this is on another level.)
Because it's possible to do hot code reloading, and since you can attach a REPL session into a running BEAM process, running 24/7 production Erlang systems - rather counterintuitively - can encourage somewhat questionable practices. It's too easy to hot-patch a live system during firefighting and then forget to retrofit the fix to the source repo. I _know_ that one of the outages in the previous job was caused by missing retrofit patch, post deployment.
The running joke is that there have been some Ericsson switches that could not be power cycled because their only correct state was the one running the network, after dozens of live hot patches over time had accumulated that had not been correctly committed to the repository.
You certainly can forget to push fixes to the source repo. But if you do that enough times, it's not hard to build tools to help you detect it. You can get enough information out of loaded modules to figure out if they match what's supposed to be there.
I had thought there was a way to get the currently loaded object code for a module, but code:get_object_code/1 looks like it pulls from the filesystem. I would think in the situation where you a) don't know what's running, and b) have the OTP team on staff, you could most likely write a new module to at least dump the object code (or something similar), and then spend some time turning that back into source code. But it makes a nice story.
That's part of it yeah. But, at least in my experience, that tells me you pushed code (to disk) and didn't load it. You could probably just notify at 4 am every day if erlang:modified_modules() /= []; assuming you don't typically do operations overnight. No big deal if you're doing emergency fixes at 4 am, you'll get an extra notification, but you're probably knee deep in notifications, what's one more per node?
But, that's not enough to tell you that the code on disk doesn't match what it's supposed to be. You'd need to have some infrastructure that keeps track of that too. But if you package your code, your package system probably has a check, which you can probably also run at 4 am.
Thank you for this post and I'll add a note for people who are seeing this and are maybe discouraged about learning Erlang/OTP/Elixir.
I generally agree with you that learning Erlang stuff can be daunting.
I will say that many things worth doing are not easy! Erlang and the whole OTP way of thinking is tough to learn in part because it is genuinely different enough from everything else that is out there that one's odds of being familiar with its conceptual underpinnings are low.
If you have trouble learning Erlang (and OTP specifically) it's not because you're dumb, it's because Erlang is different.
Learning Erlang is not like learning any other dynamic language you've learned. Learning Erlang is closer to learning a bespoke operating system designed to build reliable low-latency long-running systems. It's a larger conceptual lift than going from one dynamic OOP language to another dynamic OOP language.
It also took me quite a bit of time to understand OTP. In fact, I had to have a project that actually required what OTP offered to really get it.
Two things that definitely helped me understand were reading the somewhat-dated-but-still-useful material on the topic in Learn You Some Erlang, as well as reading through the "OTP Design Principles" section of the Erlang System Documentation.
1/3 of hn posts (maybe more) are "look at this thing we built!" or a combo of that with "plus all this VC money!" where op has basically re-invented something that has existed in Erlang since forever. I don't mind all these cool new things (it's why I visit). But personally, I prefer to cut to the chase and just use Erlang (well, in my case - Elixir).
Feels like Rust has stolen a lot of Haskell thunder, in the sense you can write similar code and satisfy that functional programming itch in a much more popular language, while falling back on imperative programming if you really need it.
I haven't been following closely, but my impression is that the dependently-typed languages/theorem-provers have stolen some of Haskell's "most purely functional, most strongly typed" hype-thunder even though they're even further from being a mainstream implementation language for most projects.
Unfortunately, I have to disagree on the absolute notion, that you will never see someone being not nice in the Erlang world. OK, you technically said "behind Erlang", not "in Erlang ecosystem". But in the Erlang ecosystem and related languages I've had at least 2 encounters, that were not nice. One outright called lack of good documentation for some projects in the Erlang ecosystem bullshit in a public chatroom, and it killed all my motivation to continue exploring that part of the ecosystem, the other one probably without knowing gave low effort dismissive responses to questions about how to best do something, without even knowing the context in which my code appeared, mixing in their personal taste and treating me like a little child.
So there are definitely unpleasant people around, just like in many ecosystems. Maybe less so, but I don't know about that. The question is only whether one lets that keep one from doing whatever one came to do.
Erlang gets a lot of stuff right for scalable web based stuff, and even though there are many of its influences that have by now made it into other languages and eco systems it is still amazing to me that such a well thought out system is run with such incredible modesty. You'll never see the people behind Erlang be confrontational or evangelists, they just do what they're good at and it is up to you whether you adopt it or not. And that is an interesting thing: the people that are good at this are writing code, not evangelizing. If I had to reboot my career I'd pick this eco system over anything else, it has incredible staying power, handles backwards compatibility issues with grace and has a community that you can be proud of joining. Modest, competent, and with a complete lack of drama.