The author doesn't talk a lot about preemptive scheduling, which is probably the best thing about the Erlang virtual machine. This video explains what preemptive scheduling is and why it is extremely useful: https://www.youtube.com/watch?v=5SbWapbXhKo
First, the speaker creates an application endpoint that runs into an infinite loop on invalid input. Then, he shows how that doesn't block the rest of the requests. Using native BEAM (EDIT: BEAM = EVM = Erlang Virtual Machine) tools, he looks for the misbehaving process (the one running the infinite loop), prints some debug information and kills it. It's pretty impressive.
Another great (and lighter) resource, is "Erlang the Movie" (https://www.youtube.com/watch?v=xrIjfIjssLE). It shows the power of concurrency through independent processes, the default debug tools, and the power of hot code reloading. Don't miss the twist at the end.
I completely agree. I used to be a big believer in monad based concurrency using Promise/Future/Async/etc. But the simplicity of using preemptive, user-space scheduling has changed my mind. It makes concurrency, especially IO concurrency, cleaner. There are fewer moving parts in your code because the VM is doing a lot of the heavy lifting for you.
(not disagreeing, but having watched the past few years as "threads bad, don't know why though..." it is amusing to see thread-style concurrency rediscovered with a new name).
This quote from Joe Armstrong explains the difference better than I can. There are other factors than just this in play of course.
> [Erlang] is a concurrent language – by that I mean that threads are part of the programming language, they do not belong to the operating system. That's really what's wrong with programming languages like Java and C++. It's threads aren't in the programming language, threads are something in the operating system – and they inherit all the problems that they have in the operating system. One of the problems is granularity of the memory management system. The memory management in the operating system protects whole pages of memory, so the smallest size that a thread can be is the smallest size of a page. That's actually too big.
> If you add more memory to your machine – you have the same number of bits that protects the memory so the granularity of the page tables goes up – you end up using say 64kB for a process you know running in a few hundred bytes.
Great point! However, I've found that the implementations of user space, preemptive threads (UPT) have friendlier APIs than the implementations of kernel space, preemptive threads (KPT). Also, UPT APIs generally come with really nice synchronization primitives (like STM in Haskell) or designs that remove the need for them (like message passing on BEAM). Also, I really like not worrying about creating too many threads and having to use a thread pool because the overhead of UPT is generally so much lower.
That is like saying queues are basically the same as lists!
size, scheduling and the lack of shared state add up to vast differences. Also the message parsing primitives are way more lightweight than the thread based equivalents.
A Monad is any data structure with the follwing two methods:
(>>=) :: m a -> ( a -> m b) -> m b
pure :: a -> m a
the monadic bind is exactly `andThen` in javascript:
(>>=) :: m a -> (a -> m b) -> m b
(>>=) :: Future a -> (a -> Future b) -> Future b
Future.andThen :: Future a -> (a -> Future b) -> Future b
and the monadic `pure` is exactly the `Future` constructor:
pure :: a -> m a
pure :: a -> Future a
Future.always :: a -> Future a
Every Monad is a Functor, so you also get this one for free:
map :: (a -> b ) -> m a -> m b
-- if I know how to convert a's to b's I can convert a future a to a future b
Future.map :: (a -> b) -> Future a -> Future b
So basically, saying "futures are a monad" gives you all the useful functions that you would expect from a future based API, which is nice. It means you can use all kinds of higher level functions not specific to futures to abstract behaviour:
Like, given a list of element,s and a way to create a future for each element, create a future that returns a list of elements:
forM :: (Monad m) => Array a -> (a -> m b) -> m (Array b
Future.forEvery :: Array a -> (a -> Future b) -> Future (Array b)
The notation in the GP is Haskell-ish type notation, which is certainly different to the majority of languages out there today, but it's actually fairly simple (difficult to learn, but simple).
Quick glossary:
:: Defines a new function signature. The left hand side is the function name, and the right hand side is the signature.
a -> b A function which converts something of type a to type b
m a A generic m of type a. In Java/C# this might be written as m<a>. eg, `Array String` means an array of string elements, and might be written in C# as Array<String>.
Types starting with a lowercase letter (m, a, b) generally mean the type is generic - as in the function will take any type.
It might be helpful to define a function you already know:
string_to_int :: String -> Int
Or, for changing an Array of something to an Array of something else:
map :: (a -> b) -> Array a -> Array b
Here, we're defining a function that:
1. Takes a function that converts an 'a' to a 'b' (a -> b)
2. Takes an array of 'a'
3. Returns an array of 'b'
Notice that 'a' and 'b' can be anything! Since the first parameter* is a function that handles the conversion from a to b, the map function actually doesn't need to know what type a and b are.
Let's say we pass in the `string_to_int` as the first parameter, now the type checker will actually infer the following function type:
That's the Haskell language. There are a lot of "big" concepts in the language itself, compared to just about any other language from the last 50 years that people get paid to write, but the notation isn't hard to straighten out after a few tries. There's a really good intro here: http://learnyouahaskell.com/
They are just a way to wrap up the code/data somewhat similar to how callbacks are used in Javascript async code. And from using Haskell I believe they are an excellent abstraction when doing any async programming, once you wrap your head around them.
Although in terms of concurrency I've found Elixir/Erlang actor-esque process based system with messaging much more intuitive when building complex programs and much simpler to reason about than the various solutions Haskell offered. But I'm also not an expert at Haskell so take that comparison with a grain of salt.
Many of the actual implementations of these types are not actually monads by the monad laws, but they are monad-like in that they are containers for execution that can be composed with each other.
I have just started reading Elixir in Action[1] by the same person in the video. I've been enjoying it so far, I think things like the stuff demonstrated will be in the book.
Preemptive scheduling is great, but there are some times when it is bad. For example, I was having problems in my Elixir code for live streaming that it was taking too many reductions to fully deserialize and process the incoming message (nothing major was in this code path, just a parsing the TCP packet and unwrapping the different layers). This ended up causing the process to get (what seems to be) overbalanced and I ended up at having delays transferring 5+mbps video through it.
I broke the deserialization up into 3 separate processes (1 for handling the I/O, 1 for packet deserialization, 1 for message deserialization) and while this fixed all the issues (presumably because it can balance the processes more effectively since no single process went over the reduction limit) it turned the effectively synchronous parsing process into a split async process and caused a lot of extra complexity around that.
I'm sure there's more to what caused the issue than just reduction limits but it also made me wary about the magic happening under the hood.
Not just scheduling — the ability to forcibly terminate processes itself is a key part of why Erlang can be resilient to failures.
It's a key area where I think Go went wrong. Without the ability to kill a goroutine, you can't implement caller-enforced timeouts and canceling — anything that should be able to be canceled has to explicitly check for some signal sent by the controlling party. This is why Go ended up with Context, which infects everything it touched.
And you also can't create Erlang-style supervisor trees; you can't create self-repairing hiearchies of control. And you can't do things like use a debugger to attach to a process and terminate runaway goroutines.
You may have been able to bump the process priority to give that process more scheduler time if it's a critical process in a hot path. Perhaps you tried that though. Definitely have to be careful with it, but always an option to keep in mind!
I haven't tried that, but I'd have to make sure that every inbound and outbound RTMP connection had high enough priority to correctly execute, and that seemed like a lot of tweaking that had to be done in order to maintain the constant flow of video throughout the app with low latency.
At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly (not just with that but other potential issues I had in the back of my mind as well).
> At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly
From someone with zero experience in building concurrent systems: isn't this true for every sufficiently complex system? If your system is complex enough, then no general framework will help much, right?
At some point sure, but I can get concurrent systems up and running pretty easily in Go, C# and Rust without going too deep into the underlying details of the asynchronous framework. Erlang's VM does a lot more than any of those frameworks or runtimes do though because of how it handles per process garbage collection yet globally shared binaries, the preemptive scheduler and how it balances process load, the hot-reload capabilities etc... It does a lot of things really well but there's a lot of magic behind it, and I seemed to brush against that much faster than I have in other languages.
Do you mind me asking what kind of throughput you are seeing for a process like that where you unpack video across some processes? I'm working on something similar using Go and it'd be great to know if Elixir is a good alternative or not.
I know of some Erlang based platforms that achieve 10gbps throughput. I hadn't done exhaustive testing on my prototype as I've changed gears away from my home-grown solution to modifying some open source C++ media servers that are a bit more proven than my own stuff.
Isn't pre-emptive multitasking the same strategy used by OS threads etc. where some global scheduler decides which tasks are running? I suppose the VM is managing the threads here rather than OS, so "green threads" like Go. Regardless, it would be pre-emptive multitasking in any multi-threaded Java program or Go program. Just trying to understand what the "secret sauce" of the Erlang VM is.
Is it just that context switches are a lot cheaper within green threads rather than the OS thread?
Pre-emptive scheduling is indeed the approach used by most operating systems (those of us who are bit older will remember when the most common version of Windows was co-operative scheduling - that was fun, any single application misbehaving could hold all the resources on the machine and that happened quite a lot!).
The genius of Erlang is that instead of a per machine it was designed to be distributed from the start so you have a lot less inter dependencies when you want to scale across machines (or cores since that's where Erlang/Elixir shines at the moment), that was one of the design goals since it was designed for the kinds of software where that was a desirable quality (back when hardware was way slower than now), dropping a single call/circuit wasn't the end of the world if the rest stayed alive.
Lifted from the wikipedia article :-
Everything is a process.
Processes are strongly isolated.
Process creation and destruction is a lightweight operation.
Message passing is the only way for processes to interact.
Processes have unique names.
If you know the name of a process you can send it a message.
Processes share no resources.
Error handling is non-local.
Processes do what they are supposed to do or fail.
I think when you get down to it there isn't really anything you can do in Erlang you couldn't do in another language, at one time Erlang went through C as a build step, it's that it was a fairly radical departure as a solution to a problem at the time (telephony) that also happens to fit a fair few problems we run into on the internet.
I'm not an Erlang/Elixir programmer (so I've no dog in the fight) but I've been fascinated by the language since I read about it in the early 90's in a programming journal, it seemed alien then and it seems alien now (much like APL).
Amongst my programmer friends Elixir seems to be really popular with Ruby programmers (I'm sure there is a reason but I'm not a Ruby programmer either so I couldn't tell you).
The author says it's "hard to use Elixir with services like Heroku because your instances won’t find each other by default (the way they’re supposed to)".
I just wanted to mention https://gigalixir.com which is built to solve exactly this problem.
We're considering to become your customer, but currently our backend isn't only Elixir but also a nodejs app that mostly prerenders our React app. How do you suggest people combine Gigalixir with non-elastic services or, say, databases?
For databases, you can provision one through Gigalixir, or you can provision one yourself with Google Cloud SQL in the us-central1 region.
For other apps, anything running in GCP us-central1 will be in the same high latency network, but some customers interop with AWS or Heroku just fine albeit with slightly higher latencies.
It's obviously a lot more work but since AWS has the biggest market and mind share, a presence in AWS will be a huge boost for you. Also make it clear to customers that you support both GCP and AWS because low latency access to database is a huge concern for most applications.
Thanks, clear! Any chance for a EU-based region soon? Most of our customers are in the EU and some are required to have their data inside the EU at all times for legal reasons.
I'd consider using gigalixir in the future if I have a need to host/run a personal+stateful app. Looks neat. Will have to check it out sometime. Bookmarked.
Right now, almost everything is done from the command-line interface. We're building a web interface, but we wanted to be command-line first so it integrates with scripts and tools.
All of my pet projects are run on Elixir/Phoenix. If there is a language/framework to learn, this would be it. As approachable as Rails with the benefits of handling real time applications (chats are trivial with Channels) and faster runtimes than Rails (by orders of magnitude).
Happy to help anyone out if they're interested in learning!
Serious question, though I know it will sound negative... I really don't like Rails and don't like Ruby or VB.NET, either (syntactically), which means Elixir is off-putting for me, at least in how it's been marketed. Do you know of anyone like me who has come to Elixir and liked it?
Note: I like Clojure, F#, and other functional languages, so that's not a hurdle.
I learned ruby on a big project about...5 years ago. I've been using it ever since at work and I definitely understand the pain points. With Ruby you really have to learn to understand and appreciate where its strengths are to be productive with it (how close it can bring you to Aspect Orient Programming is probably the biggest).
The only similarity that Elixir honestly has with Ruby is clean syntax and general emphasis on productivity. Everything else is different. Same with Phoenix. On the web side of things it has routes, controllers and views...but there's no magic involved. It's all very clear, explicit and flexible in how it does things. The view layer is honestly one of it's biggest strong points because the way it's handled is the core reason for all of those "My site was faster without caching" articles...but you don't have to work differently to enjoy those gains.
Big Nerd Ranch did a solid write up on the ins and outs of why.
Different strokes. One of the big Phoenix perks is that using it instead of just raw Plug has very little performance drop off. It's not a Rails vs Sinatra situation.
The Phoenix Channels implementation for websockets and the view layer for web requests are pretty phenomenal IMHO.
There are some unique bits too. Phoenix Channels is a beautiful, shit-simple pub/sub abstraction on WebSockets and provides both JS and Elixir libraries. And then came Phoenix Presence, which can track shared state (like a user's online status) in a durable and fault-tolerant way.
Both are useful libraries, but the fact that they work in server clusters as well as single-node deployments is the real kicker. That's just how Erlang rolls.
That's just genservers though right? All of this stuff is just Elixir and generally can be done in Elixir in a few lines of code.
We're experimenting with a truly OTP driven web application right now(No API) where we connect the React app directly to the Elixir backend and only persist to a database on log out. Everything else is persisted in memory and run through genservers which are self healing and self hydrating.
I came to Elixir from Clojure and was not a big fan of Elixir's syntax, but I've found that after a while it mostly fades into the background (much in the same way that Clojure's parens do).
The language and Erlang's virtual machine are generally a joy to work with, and I have no regrets pushing past the syntax.
(If you're asking about what flavor of Javascript I use) I just use plain JS for the bits of JS I need, but nothing I have been working on needs a heavy front-end.
There's not really an Elixir equivalent of Clojurescript to Clojure (there is an Elixirscript, but it's a work in progress and I'm curious whether it can ever capture the nice experience of using the Erlang VM). I've seen interest in the Elixir community in Elm and Bucklescript(OCaml) and other compiled-to-js languages, but I think a lot of people just use JS.
I know you weren't asking me, but Elm on the front end, communicating to Elixir on the back end, has made my life so much nicer. I loathe JavaScript, and this Elm/Elixir combination has made me like web programming again.
I also come from Clojure and Erlang and, tbh, couldn't get into Elixir for various reasons (I've followed the development closely since its inception):
* Most of the Elixir community comes from the Rails world, so most of the libraries focus on the web.
* Coming from the Ruby world, a lot of these new developers tend to be ignorant of the underlying Erlang machinery, believing that Elixir is the magic that makes Erlang a "modern" and usable language, which is simply not true.
* The tendency to favor frameworks over libraries.
* I'm probably alone in this one, but I find the syntax off-putting. It adds lots of sigils and complexity to Erlang's very succinct syntax.
* In the end, I think Elixir adds very little besides the excellent tooling to the Erlang ecosystem.
> Coming from the Ruby world, a lot of these new developers tend to be ignorant of the underlying Erlang machinery, believing that Elixir is the magic that makes Erlang a "modern" and usable language, which is simply not true.
This sounds very subjective, and doesn't agree with my experience. I don't know a single Elixir dabbler that doesn't give Erlang credit where credit is due. It's just that the syntax looks like Prolog and ML got into a nasty car wreck sometime in the 1980's, and the tooling is more like a box of parts that can be assembled into various tools. That's why I use Elixir instead, anyway.
> The tendency to favor frameworks over libraries.
I know of exactly one framework in Elixir; I assume you mean Phoenix. Phoenix is little more than a few blobs of glue around other, mostly core Elixir, libraries. The Elixir community at large is quite wary of language and framework magic. The fact that lots of us did time with Rails means we've been hurt by the downsides of that approach like few others. :)
> In the end, I think Elixir adds very little besides the excellent tooling to the Erlang ecosystem.
I think your second point is unfair. It has nothing to do with Ruby but the fact a lot of people are joining the platform. It takes time for them to understand how everything fits together. Most experienced developers I talk to are actually appreciative of Erlang. Elixir also helps a lot since it points developers to use the Erlang libraries instead of wrapping them.
But mostly you shouldn't expect people to grasp functional programming, concurrency and OTP when they are just getting started.
Not only strings, but better error messages, the new breakpoint support, functions for working with collections, protocols, tasks, etc. It goes quite beyond tooling.
I have, and quite liked it. The problem seems to be that Elixir has much more momentum and better tooling (from a web-dev perspective). Am I wrong about that?
Elixir definitely has more momentum at the moment (mostly due to an influx of Rails refugees IMHO), but as someone who came over to Elixir from Erlang rather than from Ruby I think you will find that it is not hard to switch back and forth between Elixir and Erlang once you get used to it. Elixir, if you can get past the Ruby -ish syntax, lifted a lot from Clojure as far as what is happening underneath the surface and you can call back and forth between Erlang and Elixir at eny time. This lets you do things like write large chunks of your app in Erlang and then glue everything together using Elixir/Phoenix to take advantage of the larger Elixir web-dev community. Phoenix itself is based upon a lower-level HTTP library written in Erlang.
Not perfect, but it does allow you to leverage the advantages of Elixir (e.g. Phoenix and the related web-dev community) while still letting you get most of your work done in Erlang.
There is no reason to use LFE. It doesn't add anything that Erlang does not have (besides macros) and it is very poor on the documentation side.
Even if you don't like Elixir, it brings more to the table than just syntax and macros, such as tooling, Unicode support, protocols, better error messages, better debugging, tasks, etc.
If Elixir is not your cup of tea, definitely Erlang all the way.
> I have, and quite liked it. The problem seems to be that Elixir has much more momentum and better tooling (from a web-dev perspective). Am I wrong about that?
No, you're not wrong. Elixir also supports meta-programming, whereas Erlang does not. I like both languages though, primarily due to the BEAM VM. You can't go wrong with either one.
Elixir seems much more like Clojure without the parens to me than Ruby. (Yes, I know Clojure can actually use fewer parens than the equivalent Java, just referring to the Lispy syntax in general.)
Dynamically typed, functional language with immutable data structures and a strong philosophy about how to best handle concurrency and scaling.
The key difference is JVM vs. Erlang VM as the base and all that implies.
It's a functional language! And it's based off of Erlang, so if you like Erlang, you can use call Erlang directly in Elixir. And if you like piping things / pattern matching (common in F#), I think you'd enjoy it :)
First of all, Elixir's relationship to Ruby is only skin deep.
The languages aren't as similar as some like to suggest, in my opinion.
Personally, I like Elixir better than Clojure because memory in the BEAM VM is isolated. The JVM uses shared memory, which has caused me issues before. I also like the tooling and build tools for Elixir a lot better. Clojure is weak in this area, I think. I like Elixir's syntax much better too. For me, it feels a lot cleaner and easier to type, but that is subjective.
Regarding Rails vs. Phoenix, I cannot really speak to that. I don't do monolithic apps anymore, so I don't use Rails or Phoenix. Elixir has a library called Plug built into it that lets me easily make REST APIs for micro-services, so that is what I typically use. For me, this eliminates a lot of the issues I had with Rails.
You should hear Chris McCord (Phoenix author) talk sometime -- he is extremely wary of sprawling design in a way that might please you.
I came to Elixir from Scala but also having a Rails background. I was quite sick of Railsy magic, because I'd gone deep with the Ruby metacrap and I got to taste the other edge of the sword. Elixir was a breath of fresh air.
If you enjoy concurrent programming with nice syntax and few surprises, you might end up liking it. Elixir is basically Erlang with modern syntax and vastly improved tooling.
The syntax is bad (just like ruby), and there are some annoying limitations with macros, but the semantics are very good, and the ability to keep thousands (or hundreds of thousands, or even millions if you know what you're doing) of stateful connections to clients is very useful in practice.
The macro limitations are necessary to maintain solid concurrency. Yes, it can be annoying, but much less so than trying to track down an intermittent concurrency bug.
Superficially, `do ... end` blocks look like English, because they are English words. On the other hand, they don't really correspond to any real grammar construct of the English language. They live in the the uncanny valley where you try to read it like English, but you can't actually read it as such. For some reason, the use of keywords like `def`, `defmodule` or `defmacro` doesn't bother me as much. Maybe because they are not block delimiters?
In my opinion, braces, delimiters or parenthesis create less visual noise than endless cascades of `end` keywords:
def f(x) do
...
end
end
end
end
There are other block keywords besides `end`, which serve no purpose other than simplify some keyword list arguments at the cost of higher complexity.
In general there is too much syntax sugar for things that don't really benefit from it.
Parenthesis in function calls are optional in some places but not others, which creates some unnecessary confusion.
The syntax for anonymous functions is
fn arg -> do ... end
The `end` keyword is strange and I keep forgetting it (that's more a problem with me than the syntax, I know).
The language is subtly whitespace sensitive in some places, while being mostly whitespace insensitive.
Macros can't take a variable number of arguments, which leads to the proliferation of Special Forms (it's a kind of macros that is treated as special by the compiler) which could perfectly be macros if it weren't for this limitation.
For a situation where this causes problems and requires some extra weirdness in what should be a simple DSL, look at this Github issue in the new testing library scheduled for inclusion in the standard library: https://github.com/whatyouhide/stream_data/issues/21
> Macros can't take a variable number of arguments, which leads to the proliferation of Special Forms which could perfectly be macros if it weren't for this limitation.
This is not true.
We have only two variadic special forms: "for" and "with".
"for" needs to be implemented as a special form as it emits some optimizations that are only available at the Core Erlang level.
"with" needs to be implemented as a special form as it has different lexical properties than a bunch of nested cases.
None of those could be implemented with macros.
Regarding whitespace sensitiveness, most languages have subtle issues, to varying degrees but especially if you have optional line terminators. Although it is true one or two extra cases appear in Elixir due to optional parentheses.
> "for" needs to be implemented as a special form as it emits some optimizations that are only available at the Core Erlang level.
That's interesting. I wonder if a future Elixir version could have a way of defining macros which sould emit these optimizations. Is this a likely direction for future developments?
> "with" needs to be implemented as a special form as it has different lexical properties than a bunch of nested cases.
Could you expand on this a little? What are those different lexical properties?
> That's interesting. I wonder if a future Elixir version could have a way of defining macros which sould emit these optimizations. Is this a likely direction for future developments?
To do so, we would need to compile to core, and that means tools like Cover and the Erlang debugger would no longer work with Elixir. It is unlikely we will go to this direction.
> Could you expand on this a little? What are those different lexical properties?
If you compile a "with" to a bunch of cases, a variable from the outer case would be available to both success and failure cases. Take this code:
a = true
with true <- a = false do
a
else
_ -> a
end
In Elixir it returns true because `a = false` has no impact on else. But if it compiled to a bunch of cases, we would get:
Not the OP, but I share the opinion. I strongly prefer a precise syntax over ones where parens are optional, but required in some circumstances, etc. I don't like Ruby's block syntax at all. (Incidentally, I wish Ruby had better first-class function support instead of blocks.) I don't like "end" littered everywhere. I find the `=>` for hashes/dictionaries to be annoying, given how often I use that data structure (granted, I now mostly can do `{foo: :bar}`).
There're other things I don't love, but that's a good chunk. Philosophically, I prefer explicit code to implicit (so, Rails is out). And I prefer functional programming to OOP in general, and find that Ruby greatly favors mutable OOP. I've had a tough time navigating around the Ruby parts of most code-bases I've found, largely due to the implicit nature, and annoying language features that allow clever generation of methods in a way that makes them impossible to search for (delegate ... prefix: true for example).
You can look into drab, an elixir server-side DOM manipulation library: https://github.com/grych/drab. I've never used it, but many people in the Elixir community think it holds a lot of potential.
This project seems really interesting, I guess it could pretty much replace SPAs (and related javascript fatigue) in a lot of use cases. Am I missing some significant limitation other than moving some computation back to the servers?
The only real limitation I can see for now is a response time for the actions, which might be done locally, without requesting anything from the server. Eg, you don't want to do animation loop with Drab, for this just use the ol' good JS.
I would love to use Elixir, I find the language to be the natural successor to Ruby, given the trends in languages today. However, everything I've read says that it's pretty hard to deploy. That the cloud options are limited (or take away the advantages of Erlang VM like code hot swapping). Any progress on this front?
I think one of the difficulties here is that much of the functionality that comes from, for example, Docker + Kubernetes exists as part of the BEAM VM. A lot of the development being done today in deployment is around Docker and Docker clustering technology, which means that it often feels like you are doing things the old fashioned way when using BEAM clustering & networking technology.
There are ways to set up these systems in symbiotic configurations, like having an Elixir/Erlang/BEAM cluster that has auto-joining thanks to external service discovery. However, to use the hot-swap functionality, you are going to be working with the BEAM VM directly, which means going through the Docker container boundary.
Know that if you choose to go with BEAM and no Docker, you will still have access to amazing deployment features (like hot swapping, as you mentioned), but also clustering, service discovery, monitoring, etc. You're just going to be spending a lot more time at the command line without the pretty interfaces and the sleek modern tools.
To get a better understanding of running BEAM, I highly recommend "Designing for Scalability with Erlang/OTP". You'll have to learn Erlang to get through it, but IMO it's the best introduction.
> you will still have access to amazing deployment features (like hot swapping, as you mentioned), but also clustering, service discovery, monitoring, etc.
Can you elaborate on this statement? My understanding is that you can have these things (minus hot swapping) cleanly on Docker.
Yes, you definitely do. What I meant is if you didn't use Docker and just used BEAM, you would still have access to all those features that are present in Docker + clustering technology.
I've been running an Elixir app in production on about 20 nodes for 18 months. We just build Debian packages and deploy using Salt.
Additionally, most of the cluster nodes run on Spot Instances in AWS so they are relatively inexpensive also. When a new instance comes up it connects to the cluster and starts serving requests. When an instance is killed, traffic is routed to the remaining nodes. Works great.
Do you have a blog post or anything that explains this process? How many nodes in the cluster? Why did you build Debian packages? How do nodes join/leave the cluster (DNS?)? Do you use ELBs at all?
I really should write a blog post about it :) but right now there isn't one.
Debian packages are easy to work with and we keep them as our primary artifact for any given release.
The number of cluster nodes varies between 10 and 35 depending on the workload at the time.
Nodes join the cluster by contacting one of the three "static" nodes that are not spot instances. If they can talk to at least one of those they get knowledge of the entire cluster. When one of the spot instances is going to be killed there is a notice is the instance metadata. The nodes just watch for that and initiate a normal shutdown if they see it.
Yes we do use ELBs but just as TCP proxies and for TLS termination.
My first ever docker build was a multi-stage elixir compilation. The end result is a docker image that can be deployed to Heroku or k8s that works great. It's not that hard to get a good docker setup for it. It's even easier if you use Mix and not releases (like I did). However, releases are cool.
Code hot swapping is not really docker friendly (obviously). I haven't done that, but work's general philosophy has been to have recoverable processes such that starting up the app will bring up the correct processes loaded with the right data. However, it's definitely possible to setup hot swapping but is going to be a learning curve to it.
Code hot swapping is not particularly friendly at all. It's impressive that you can, but the time required to write the code carefully enough and QA the upgrade doesn't seem to be worth it for most applications.
Sure, a telco switch that has to be up 24x7, hot swapping makes a lot of sense. Most applications can spare the brief downtime to reload the old-fashioned way (or have multiple servers to allow upgrades to be done without downtime).
Code hot swapping can often trivially be done (by simply compiling and copying the fixed module, then 'l(module)' in the shell) if the module does not change the state record or break function signatures. I have used this countless times to fix a module function with no downtime. You don't always need the full code upgrade procedure, which is indeed far more onerous.
I agree, I should have been more specific. Individual module upgrades are indeed straightforward. That didn't sound like what the poster was describing.
One new service that tries to make Elixir deployment easier is https://gigalixir.com. If Heroku was built with only Elixir in mind, this would be it. Disclaimer: I'm the founder.
Absolutely! I'd love all the feedback I can get. You can leave it here or you can email me directly here (base64 encoded): amVzc2VAZ2lnYWxpeGlyLmNvbQ==
It's not very hard to deploy. Just use the proper tools to compile and build a release (on a VM emulating the same architecture as your server), rsync the resulting files into the server and strat the app.
Getting hot code reload to work is harder, but for a "normal" basic deployment it's just this. App restarts take less than one second of downtime for my app, and that's a cost I'm willing to pay.
You shouldn't do this by hand of course. Write an Ansible playbook or something else that does this for you
"Hard" is relative, of course. For small teams without a release engineer, options like Heroku, now, etc. are much easier than what you are suggesting.
Deploying to Heroku is relatively easy, the only downside is you lose access to clustering (plus hot code upgrades but most people don't use that anyway). There's great documentation available for heroku as well:
There are already recipes for deploying to Heroku, and I've had no problem using Elixir and Phoenix apps on Heroku for the past 2 years. If you use EngineYard or other bare-metal solutions like AWS or Google Cloud, I would imagine there's something you can glean from that.
replace pet with nothing and you get my life.. it's also FAR cheaper to run at scale than ruby.. which means you can offer faster/better apps than your ruby peers at a lower price point.
as a Ruby developer from Rails 2 > Rails 5.. about a year ago I made the switch to non-monolithic applications to Elixir API(no phoenix) and ReactJS front ends.. Match made in heaven.
I don't think the Gartner hype cycle applies to Elixir, and I think that is largely because it is built on top of a very mature, production-tested platform (Erlang OTP). I have been using it in production for almost two years without issue on https://teller.io/, so if the GHC applies, it's very elongated!
I agree with you that Erlang is not at the peak of inflated expectations. The community seems to have hit a steady state where it is probably closer to the plateau of productivity. Elixir newcomers, however, often don't have experience with Erlang, so the expectations might be all over the place. Hopefully, since Elixir is built on top of a proven system, it should weather through the trough just fine.
Elixir and Phoenix exceeded all my expectations. I thought about what I wanted to build, read the docs and built it. Done.
When I was still using python, I spent days (weeks?) reading about the problems, looking for design patterns that might work for my use case, and finaly quit in frustration.
So you use Elixir on your finance app? I was wanting to use Elixir on a payment processing app... any details with your experience running in production you care to share?
Seconded. I've been trying to find good write ups on production deployment best practices, especially people who have used it commercially for over a year.
The only part of that article I'd clarify is around deployment and devops best practices.
You can deploy Elixir exactly the same as any other language. In some cases, it just means making a decision that you don't need some of the extra features that are available like hot reloading...which every project doesn't need.
You can still use immutable images and take advantage of the built in distributed databases by using mounted block storage if need be.
You can use everything out there. Early on figuring out some of the port mappings and how to handle things was more difficult but as far as I've seen, those problems have mature solutions all around now.
I'm less interested in what you can do, and more interested in which techniques have emerged (or are emerging) as the best practices. For instance, if I want both hot reloading and deployment on AWS, what's the best way to configure and run this? Etc. Are you aware of any devops-for-Elixir focused resources?
The only reason I say that is that every project doesn't need hot-reloading. It's one of those things that is only recommended where you need it.
Just think of it in the same way that you don't need every available jar file for a Java project. The capabilities of the system are expansive, but not always called for.
Distillery is the main deployment related library out there for Elixir and reduces hot reloads to a an argument on the build.
Hot reloading also adds a ton of hidden complexity to your projects, since you essentially have to be very careful about every module boundary (as my understanding is that every time code goes from one external module to another non-OTP module it uses the new version of that new module, and thus any non-backwards compatible signature change will crash the process). You can't really casually support hot reloading and need to fully commit to it, unless you don't care if things fail when you hot reload at which point you might as well just do a full reload and keep confidence in the system.
I have not started looking into phoenix as I'm still exploring elixir, but I'm happy to have started learning elixir with koans along with official elixir guide.
Great write up! Guys, if you're on the fence about learning Elixir - dive on in!
You won't be disappointed and you'll be surprised how many times you'll want to reach out to auxiliary services and find out "Oh I can just use ETS/OTP/GenServer/spawn".
I don't like the debugging workflow described in the linked article "Debugging techniques in Elixir", it reminds me of DDD: a separate tool not integrated with my main development environment and requiring extra manual steps. I tested both the Jetbrains plugin and the vscode extension, both failed (unsupported versions, bugs, etc...).
To elixir users: what do you think about the state of debugging tools? What is your workflow?
Well you have the basic IO.inspect/2 that you can use for "printf" debugging. It works really well with the pipe operator because it prints the structure and returns the structure itself not the string representation, so you can do stuff like:
Also with Elixir 1.5 (OTP 20) you can set breakpoints easily from within IEx.
There is also IEx.pry where you can set the "breakpoint" in the code itself.
There is the observer tool where you can inspect each process and see numerous information that can be useful. When in IEx just type :observer.start and try it out yourself.
I suggest reading Designing for Scalability with Erlang/OTP, particularly the chapter about Erlang/Elixir tracing facilities that can be really useful when debugging live production systems.
Compared to other languages, I find that I need less debugging in general. When I need debugging it's usually because of libraries/external dependencies with weird behavior, but sine everything is a function, it's easier to figure out what's wrong and where since you can call anything as a function in iEX.
Yes Elixir handles concurrency better than Ruby. In terms of raw performance it's nowhere near Go, Java, C++ though. Rails/Django are fast enough for almost all apps, if you need the additional performance improvements of a faster language you'd probably end up with one of those 3. Wonder how much need there is for a language that takes the middle ground in terms of performance. Looks very sexy though, really want to build something with it :)
> Rails/Django are fast enough for almost all apps...
Absolutely not true for Rails. In an Elixir app with Phoenix my response times are anywhere from 0.2 to 12 ms while the Rails app that uses the same database has response times anywhere from 300 to 2000 ms.
(Before you ask, I did went the extra mile to rewrite one of the small Ruby on Rails apps from my job to Elixir & Phoenix and maintain it as a feature-complete clone, just so I can have objective data.)
If we compare web app stacks, Elixir & Phoenix app is orders of magnitude faster than Rails in particular and this is not an exaggeration -- can't talk for vanilla Ruby though, I've had positive experiences with mid-sized Sinatra apps; but they still weren't capable of more than 20-30 requests/sec if you are looking for consistent throughput.
I truly like Go but to me it has always been a much better choice for highly performant microservices. I tried 3 separate web frameworks 12-16 months ago and I simply concluded Go isn't a good fit for a full web app -- at least compared to the conveniences and dozens of other advantages of an Elixir & Phoenix app. Go is definitely better in several other areas.
TL;DR -- Rails is very far from "fast enough". It's very okay for internal apps used only in business teams where the load won't ever be more than 20 requests a minute.
Middle ground sounds a lot like median. Once you factor in concurrency I would suspect Elixir/Phoenix is much closer to those three than to Ruby/Rails in terms of realized performance for the resources. This is assuming a pretty vanilla MRI based solution on the beaten path.
Also yeah, the sexy; I get the sense that Phoenix may be a better platform straight up..
I think the discussion around deployment may have been unnecessarily tainted by their experience using edeliver - it's an automation layer for building and deploying releases, but as mentioned it is configured via shell scripts and because it does a lot, a lot can go wrong.
The basic unit of Elixir (and Erlang for that matter) deployments is the release. A release is just a tarball containing the bytecode of the application, configuration files, private data files, and some shell scripts for booting the application. Deployment is literally extracting the tarball to where you want the application deployed, and running `bin/myapp start` from the root of that folder which starts a daemon running the application. There is a `foreground` task as well which works well for running in containers.
My last Elixir gig prior to my current one used Docker + Kubernetes and almost all of our applications were Elixir, Erlang, or Go. It was extremely painless to use with releases, and our containers were tiny because the release package contained everything it needed to run, so the OS basically just needed a shell, and the shared libraries needed by the runtime (e.g. crypto).
My current job, we're deploying a release via RPM, and again, releases play really nicely with packaging in this way, particularly since the boot script which comes with the release takes care of the major tasks (start, stop, restart, upgrade/downgrade).
There are pain points with releases, but once you are aware of them (and they are pretty clearly documented), it's not really something which affects you. For example, if you bundle the Erlang runtime system (ERTS) in a release, you must deploy to the same OS/architecture as the machine you built the release on, and that machine needs to have all of the shared libraries installed which ERTS will need. If you don't bundle ERTS, but use one installed on the target machine, it must be the same version used to compile your application, because the compiled bytecode is shipped in the release. Those two issues can definitely catch you if you just wing a deployment, but they are documented clearly to help prevent that.
In short, if there was pain experienced, I think it may have been due to the particular tool they used - I don't think deployment in Elixir is difficult, outdated, or painful, but you do have to understand the tools you are using and how to take advantage of them, and I'm not sure that's different from any other language really.
Disclaimer: I'm the creator/maintainer of Distillery, the underlying release management tooling for Elixir, so I am obviously biased, but I also suspect I have more experience deploying Elixir applications than a lot of people, so hopefully it's a wash and I can be objective enough to chime in here.
I don't know why, because I don't think they are all that similar, but I'm often asked to defend my choice of using Elixir professionally vs. Go. From the article, this is one of the big reasons I chose Elixir over Go:
"Go’s goroutines are neither memory-isolated nor are they guaranteed to yield after a certain amount of time. Certain types of library operations in Go (e.g. syscalls) will automatically yield the thread, but there are cases where a long-running computation could prevent yielding."
goroutines also take up about 10 times the memory.
It's true, but would very rarely come up in practice; it's an extremely simplified statement. Any function call(that isn't inlined) is an opportunity for the scheduler to preempt the goroutine. I has been described as partially preemptive.
I thought an erlang process takes up at least 309 words of memory? That would make it <4x on a 64 bit system?
Erlang has its own HTTP(s) servers, like cowboy (https://github.com/ninenines/cowboy) so it doesn't need Nginx and the like. Those servers simply take the place of Nginx. There is nothing special about Nginx that you can't copy in other languages, like Erlang.
The main web framework for elixir, Phoenix, is just something that bundles cowboy (the webserver) with some utilities on top of it. No need for Nginx.
I do use Nginx in my side projects in front of my Elixir web app, but that's because of some conveniences Nginx brings (having Nginx is probably making my requests a little slower).
It's just Nginx + Phoenix on a rented Linode server. It's in beta, so I don't care much about security beyond the basics...
EDIT: I could drop Nginx and use "raw" cowboy (the webserver behind phoenix), but Nginx makes it easier to setup TLS certificates and I'm lazy enough to keep it around just because of that...
Wait a minute. In the graph in the article you shared, it says that instead of storing persistable data in Redis/MongoDb, you would use Erlang. Do people seriously not use a database in Erlang?
You have persistent processes which you can send and receive messages from which can be treated like a data structure store, as well as ETS if you need more advanced querying and Mnesia if you need embedded persistence.
I'm not sure how these compare to Redis in terms of performance/latency but Mongo is more of a "proper" database than any of these.
Ok. I've used Elixir and read Thomas' Programming Elixir, and from the little exposure I got I knew it was normal to use it as a kv store but still use something like postgres for a traditional db. I did a double take when I saw Mongo in the table.
We definitely use databases such as Postgres, MySQL, Riak, etc. It is just that when it comes to ephemeral data, the tools that ship with the Elixir and OTP may be enough. See how Phoenix does the whole pubsub and presence without Redis or similar.
One thing I didn't see covered that I'm currently trying to understand with Elixir is the relationship between process state and disk-backed state. (For example fetched via Ecto.) Does the role of a traditional RDBMS change in an elixir system? What are the durability guarantees of process state? Etc. Any real world experience would be super helpful to hear about.
If you don't do a `kill -9 <your_app_pid>` then any of your processes inside will be restarted with last-known good state which the Erlang VM (BEAM) keeps internally. It's not backed by disk state. It's a VM-wide guarantee, so to speak.
The official Elixir tutorial has an OTP chapter in which you are encouraged to actively kill processes and observe them being restarted with good state in real time:
First, the speaker creates an application endpoint that runs into an infinite loop on invalid input. Then, he shows how that doesn't block the rest of the requests. Using native BEAM (EDIT: BEAM = EVM = Erlang Virtual Machine) tools, he looks for the misbehaving process (the one running the infinite loop), prints some debug information and kills it. It's pretty impressive.
Another great (and lighter) resource, is "Erlang the Movie" (https://www.youtube.com/watch?v=xrIjfIjssLE). It shows the power of concurrency through independent processes, the default debug tools, and the power of hot code reloading. Don't miss the twist at the end.