
Elixir at PagerDuty - romanhn
https://www.pagerduty.com/blog/elixir-at-pagerduty/
======
slezakattack
As an Erlang developer for the past couple years now, I love seeing the
adoption and excitement around Elixir and the BEAM. I will admit I always
shudder when people very quickly call out on Erlang's syntax as a reason not
to use it. Feels like a pretty lame excuse...

All that said however, it kind of bugs me when I see posts like this (no
matter the language) that go somewhere along the lines of "I managed to
introduce language X and it was all rainbows and unicorns! Everybody loves it
and work is fun again". Like, you had absolutely no problems with it? In my
experience, people don't immediately grasp the actor model or how to manage
processes (i.e. should I spawn a process for a short-lived task? should this
monitor another process? should I link? should it need supervision?).

Given that you're basically changing a methodology amongst your devs, it can
be expected that things won't get written "right" the first time. I'm not
referring to little idiosyncrasies like coding convention or best practices
but literally misunderstanding of gen_servers and supervisors. Sure, perhaps
transferring from Ruby to Scala to Elixir helped made the transfer less
painful given their similarities but surely there were complications?

~~~
dudul
Your first comment is kind of funny, because while Elixir's syntax may appear
more "esthetic", I find Erlang's way more consistent and logical.

~~~
OvermindDL1
Entirely agreed. Elixir's Macro's are awesome, and Erlang could definitely use
an Elixir-Macro-Like Parse Transform library (could definitely exist),
Erlang's syntax I find far more readable, consistent, and logical as well.
Elixirs syntax has a lot needless, hard-to-read, noise, but it's not hard to
overcome the noise to get the benefits of the overall ecosystem and macros.

~~~
pera
This topic is quite subjective, but could you provide some example? Overall
Elixir seems to me a bit easier to read, for instance:

 _Elixir_ :

    
    
      "Hello,How,Are,You,Today" |> String.split(",") |> Enum.join(".") |> IO.puts
    

_Erlang_ :

    
    
      -module(tok).
      -export([start/0]).
    
      start() ->
         Lst = string:tokens("Hello,How,Are,You,Today",","),
         io:fwrite("~s~n", [string:join(Lst,".")]),
         ok.

~~~
dxhdr
> This topic is quite subjective, but could you provide some example?

Elixir's pin operator is a good example of unnecessary complexity.

~~~
dqv
I'm okay with this trade off instead of having to do Variable0, Variable1,
..., VariableN. It would be cool to have an option to turn off rebinding for
those who don't want it (and then not requiring the pin).

~~~
mononcqc
Elixir already has those rebindable variable, which can be "turned off" by
using the ^ prefix on a match variable. Wouldn't that make the |> less useful
to you already?

------
jxub
My impression about the Elixir/Erlang so far has been the same.

> _Elixir comes with one of the nicest and most helpful communities around_

Using Elixir, it's easy to write gorgeous code, with a language that for the
most part is really minimalist, and you get to play with the great piece of
software that OTP is without all the warts of Erlang (which is mostly the
developer experience). The BEAM community is fantastic and really supportive
(disclaimer: I'm a GSoC student working for them on making Elixir bindings for
Barrel, a replicated document database).

Maybe we could think that it's best in terms of the character of the community
if Elixir stays small, however Rust community seems a great example of
something bigger to follow that hasn't evolved into a soulless graveyard of
enterprise code scattered on the githubs and maven centrals of the world.

There has to be a way between hand-rolled craft parsers you write with a beard
and an IPA in hand and another AbstractObservableContainer written by Dilbert
that doesn't compromise the ecosystem or contort it in any funny way.

~~~
gamache
Rust is a community of very nice and helpful people, much like Elixir but with
more money behind it. However, I certainly enjoy Elixir's unwillingness to
break working code with every release, or to force users to build against
nightly releases just to use any features from the last year. Rust is
improving in this regard, but isn't quite there yet.

~~~
steveklabnik
What stuff are you running into with this? The vast majority of the ecosystem
is on stable, but there are some holdouts. Working on them! I always like to
hear people’s pain points, it helps with prioritization.

~~~
firebacon
Just yesterday our build broke because a cargo update pulled in some
dependencies that suddenly required experimental features. I think it was
crossbeam via hyper or tokio.

Also, while I recognize it's a third party project, the hyper API keeps
changing faster than I can adapt my code. If you are aware of a more stable
HTTP server, ideally one that has proper TLS support and support for unix
domain sockets, I would be very interested.

EDIT: on the other hand, after working with rust daily for ~1.5 years, I have
only run into one compiler bug and had zero actual bugs in my application due
to language/compiler updates so far. There have been a few occasions were
essential features were missing from the standard library/language though and
several times were we had to backport some code to an older, stable rust
version that we're one (when a developer had mistakenly tested their code
locally with a newer version).

The highest up on my wish list would be that - as I understand it - you can
not currently exit the process cleanly (EDIT: early) with a non-zero exit code
in stable rust. That makes it pretty hard to implement good command line
utilities. Happy to be corrected if I am wrong on this one though.

~~~
steveklabnik
Interesting, I'll check it out: sounds like a bug in crossbeam. Thanks for
letting me know. It's also true that the language and the ecosystem are
different things; that's one reason why we're focusing on stabilizing stuff
this year, to help move people off of nightly. 1.26 contained a really huge
stabilization in this regard. It is a downside of the "put everything in the
ecosystem" approach, though.

You're writing a server with hyper directly? Is there a particular reason
you're not using one of the frameworks written on top? I may have some advice
there, depending.

~~~
firebacon
We started using Iron, but later switched to just using hyper because that
seemed easier and like a smaller API surface to target. I'm not really working
on a web application so I need zero of the routing/web features of these
frameworks.

I do need a lot of pretty specific other features though. Unix domain sockets
are currently a must (luckily that works with hyperlocal).

The code runs on a pretty resource constrained system so I like the low level
of control that hyper gives me of the request/response chunking behaviour (I
can not afford to buffer large requests in memory and what exactly I do with
the body payload differs between API calls).

One thing that would be _lovely_ is proper TLS/HTTPS support. I need support
for x509 client certificates and I need to get access to the full client
certificate data, ideally including the full chain that signed it from the
request handler. I have to admit that I currently run a hacked together nginx
in front of my app and put this stuff into HTTP headers (hence the unix domain
sockets) because I could not get it to work with rust ecosystem libraries.

EDIT: The TLS stuff is also true for the HTTP client case. Currently using the
libcurl rust binding because it's the only thing that implements all the
features I need (--cainfo, --cert, --key, --resolv). Also it needs to run on
OpenSSL or something else that supports x509v3 extended attributes (subtree
constraints).

~~~
steveklabnik
Cool cool, that makes sense. I probably don't have good advice for you then;
what I will say is that things should end up much more stable in a few months,
but there's a massive async/await/impl Trait/futures/tokio/hyper upgrade going
on, so stuff is a bit more chaotic than usual. Once that's over though, stuff
should be solid for quite a whole.

I don't know if rustls supports your use-case for TLS, but you may want to
check it out.

Thanks again. It's really invaluable to hear about these kinds of things.

------
enraged_camel
Elixir is amazing. It is everything I want in a programming language:
functional, fast, reliable and elegant.

I desperately want it to become more popular. To this end, I started writing a
book that teaches Elixir (Phoenix) in combination with Vue.js. It's still in
the early stages, but the thing I realized is that even writing the book is
fun! This was very surprising to me because I typically don't find writing
technical documentation/tutorials pleasant.

Anyway, if you're on the fence about Elixir, definitely check it out -
especially if you're coming from a Rails background!

~~~
jacquesm
Looking forward to your book!

------
peterwwillis
As someone who has managed large clusters of Erlang applications, I have to
say I am 100% convinced that the language you pick, no matter how amazing its
design principles, has no bearing on how well the application runs. If you
work for a telecom and you're actually trying to meet a nine-nines SLA, you
can do it with Erlang. But if you're a start-up throwing together some random
distributed app on a small budget, nine-nines probably isn't your goal, and so
your application won't be nearly as stable.

~~~
sb8244
Totally. Uptime & stability is a mentality and a feature, never a free ride!
At the same time, having a solid foundation and a community that knows _how_
to take it to rock solid stability is valuable.

------
everdev
> Elixir has been mostly selling itself: it works, it sits on a rock-solid
> platform, code is very understandable as the community has a healthy
> aversion towards the sort of “magic” that makes Rails tick, and it’s quite
> simple to pick up as it has a small surface area

I was a huge Rails fan back in '07 and built a number of apps with it
successfully, but I can't help feel like it's transition to a legacy framework
has already begun.

While Rails was never top in the performance category, languages are catching
up in tooling and readability. So, now you can get performance and a great
developer experience with multiple other languages like Elixir, Crystal, Go
and more.

~~~
cutler
How can you compare the overall developer experience of Crystal, which isn't
even 1.0 yet and has very few libraries, with the mature ecosystem of Rails?

~~~
everdev
You're right that the major downside I can see is the ecosystem is not as big.
But when I jumped into Rails over a decade ago, the Ruby/Rails ecosystem was
not huge yet either.

But, static typing and being able to compile and deploy a binary is a pretty
sweet developer experience. And the syntax is nearly identical to Ruby with a
few nice additions. Many companies like ThoughtBot (a huge Ruby adopter) are
jumping into Crystal. The future is definitely in compiled/static languages
IMO.

~~~
temporaryacc62
If only there were types in Elixir to get rid of those damn nulls...

Supposedly someone is making a Typescript-like precompiler for Elixir (with ML
syntax of course). We are all hoping it succeeds!

~~~
OvermindDL1
Eh, as far as I know I'm the only one doing that, and even then only as an
experiment/playground, and I don't have the time to work on it lately (have to
pay the bills after all).

It's not compiling to elixir, but the `alpaca` language compiles to the BEAM
(via erlang core I think) so it gets you close, although they are not black
boxing messages sadly.

If you know of someone else doing typing experiments on top of elixir, please
link me (my playground code is on my github repo somewhere)!!!

~~~
temporaryacc62
Found it [https://github.com/wende/elchemy](https://github.com/wende/elchemy)

The idea seems just like Typescript, a very interesting project!

------
gandreani
Thank you for sharing your experience! New languages is one of the "fun" parts
of programming. Except at work.

There are legitimate problems of legibility, maintainability, and training
when introducing new stuff into the stack. Then there's the balance that we
have to strike with re-writing old with new. How much do we gain with the new
vs how much time is it gonna cost?

Lately, these have been the things I worry about instead of algorithm design
or product design. Precisely the things we don't get taught in school or get
interviewed for

~~~
brightball
True. In my experience that largely depends on the environment.

In Java or .NET shops, there is a significant investment in the surrounding
infrastructure where the code is deployed. Has a huge impact in cost of
switching. That's before even factoring in some of the inside baseball related
to "discounts" on products like SQL Server if you have a certain number of
employees with Microsoft certs.

For just about everything else though...it's not nearly as much of a headache.

~~~
gandreani
Oh man. Those discounts sounds like such a headache. It's hard enough
considering the asks for one department already...

Environment is definitely the key. We don't use either of those languages but
we do run our site on a dozen or so servers. There's a fair bit of coupling
that can eventually add up when you run the same apps on the same servers for
years

------
jgulbronson
Awesome read! I was actually an intern on Cees' team when we started writing
db2kafka
([https://github.com/PagerDuty/db2kafka](https://github.com/PagerDuty/db2kafka)),
and got to contribute a fair amount. It was a really fun language to work
with, especially if you enjoy the Actor pattern, and had some nice debugging
tools. I'd definitely encourage anyone interested to give it a shot,
especially anyone with similar requirements as Pagerduty!

------
brightball
Great hearing more and more Elixir adoption stories from well known companies.

The quote that resonates...

> ...some of us are secretly hoping that one day, we’ll only have to deal with
> Elixir code

~~~
mercer
I've been lucky that I could choose my stack on a few projects, but I also
hope Elixir will keep getting more popular so that I can use it for the more
common contractor-type things I get offered.

------
anthonybullard
I'd love to hear more about your troubles with Scala, and any examples of
"clean code being hard to write". Do you think this had more to do with a lack
of experience with strongly typed functional languages, or more to do with
Scala's mixed paradigm?

Also, thanks for sharing your story. Elixir is a great language and I love
seeing it being adopted more and more everyday.

~~~
gamache
I'm not from PagerDuty, but I also moved to Elixir after some years working in
a hybrid Ruby + Scala shop. I concur with the author's experience that writing
clean and maintainable Scala code is hard. A few reasons:

* Haskell influence on the language, especially early on, encouraged the omission of dots and parentheses wherever possible. Just about every piece of sample code was written as an undifferentiated stream of tokens. There is a mental burden.

* Haskell influence on the community encourages category theory solutions to all problems. This harms interop with Java libraries, in addition to requiring a number of concepts that are safely ignored in many/most other languages.

* Scala's rich OO system (classes, interfaces, traits, singletons, case classes, implicits, etc) results in frequent design paralysis, not to mention taking a while to learn.

Elixir suffers from none of these problems, while retaining some of the things
that made Scala a pleasure to use, notably the actor system of concurrency and
the use of immutable data structures.

~~~
jvican
I work for the Scala Center and I'd like to comment on some of the points you
make to hopefully explain how I see things from my side. You can expect my
opinion to be biased but I'll try to stick to the facts.

> Haskell influence on the language, especially early on, encouraged the
> omission of dots and parentheses wherever possible.

This is considered an anti-pattern in the Scala community. The last library
that used this at large scala was sbt, Scala's build tool, and now everyone
discourages the use of infix operators.

> Haskell influence on the community encourages category theory solutions to
> all problems.

This is simply not true. First, there is no Haskell influence on the language.
Scala is a functional programming language, all similarities between the two
are just fundamental to how their type systems work and the foundations of the
paradigm. Second, I contest heavily that the Community encourages category
theory to solve all the problems. There are many subcommunities in Scala and
many styles, but if you talked to some Scala developers nobody would agree
that category theory should be applied to all code. In fact, most people
don't. They just use Scala because it's a better Java that boosts their
productivity.

> Scala's rich OO system (classes, interfaces, traits, singletons, case
> classes, implicits, etc) results in frequent design paralysis, not to
> mention taking a while to learn.

Absolutely agree that for someone not experienced, it may be difficult to know
what's the best way to design libraries (this is more of a problem with
libraries than applications). This is a big problem of Scala that I think
we're solving by making the language more opinionated and being more open
about what's encouraged and discouraged.

If you ask for my opinion, I think we also need to make a better job at
communicating all this knowledge that advanced Scala developers learn but that
is usually kept to closed circles.

~~~
runT1ME
>They just use Scala because it's a better Java that boosts their
productivity.

I spent three years on the 2nd largest Scala 'team' in the U.S (first as an
engineer, then leading a sub team), the problem is that using Scala as a
'better java' doesn't really buy you much productivity wise. A small
percentage of sub teams tried using Scala this way and hit a lot of road
blocks with the unfamiliar syntax, immature tooling, and other quirks.

The teams that had huge productivity gains were the ones who leaned heavily on
the functional features _WHILE_ caring about overall readability. That means
yes, we used Scalaz but we discouraged the omission of dots and parentheses
(as you mentioned), discouraged operator overloading/symbolic methods, and
were very deliberate about when to reach for features such as HKTs or some of
the more unfamiliar features of Scalaz.

Overall it lead to a very productive and fun development experience, and at
the same time illuminated many ways in which the Scala ecosystem could also be
improved.

~~~
jvican
Productivity is defined in subjective terms. I know several success stories of
companies that use Scala as a better Java and find themselves more productive.
That doesn't mean they don't do any functional programming -- it just means
they don't need to use Scalaz/cats to be happy Scala developers. The OP is
mostly concerned about Scala being a language that encourages the use of monad
transformers and catamorphisms all over the place. What I'm trying to
illustrate is that nothing could be farther from the truth.

------
vikingcaffiene
I worked on Elixir at my last job and it left quite an impression. At my
current role (c#/.net shop) I constantly evangelize for it. The fact that this
obscure language makes me comfortable enough to recommend to my entire team
speaks volumes to the quality and community around it. Cannot recommend it
enough.

------
cube2222
How is the library ecosystem with elixir? Currently I'm usually using Go where
I have a library for anything and I'm quite apprehensive to start using elixir
if there's a lack of libraries.

~~~
cnnrjcbsn
It's pretty solid. You may have to roll something yourself for really esoteric
APIs, but pretty much everything you probably want to use is supported already

------
blackrock
It seems that Elixir would be a perfect platform for Artificial Intelligence
programming.

Since Elixir is used for (1) functional programming, and (2) it uses Erlang's
OTP Server for massive parallelization.

But, I don't seem much activity in this space. What is the probability that
Elixir will emerge as the "killer app" for AI development?

~~~
rpm33
There is already some work out there since Elixir lends perfectly for
parallelization of computation and is also immutable. See:
[https://www.youtube.com/watch?v=YE0h9DURSOo](https://www.youtube.com/watch?v=YE0h9DURSOo)
and
[http://www.automatingthefuture.com/blog/2016/9/7/artificial-...](http://www.automatingthefuture.com/blog/2016/9/7/artificial-
neural-networks-elixir-and-you)

~~~
rs86
It won't never be even close to native code because of the BEAM overhead, lack
of typing, memory layout. It is suited for concurrency not computationally
intensive workloads. Not to mention the use of gpus.

------
rs86
Erlang is not remotely based off prolog. It shares the syntax because it was
first implemented in prolog.

~~~
nerpderp83
Are you attempting to confuse the conversation with facts?

------
Touche
I wonder about a language like Elixir in the age of serverless. Erlang will
always be great at the use-case it was originally written for (and what Whats
App uses it for), but that's not the scale that most startups operate in.

With serverless the advantages that Erlang has don't matter much, since the
process doesn't live along to utilize them. Instead the bottleneck is cold
start times. So these types of languages (including Java) that are meant to
start and stay running for a long time aren't a great fit. Go, Rust, possibly
Pony, are better fits. Even Node isn't a superb choice, cold starts can be
slow because of loading so many files.

~~~
qaq
You do realize that outside of cloud providers hype "serverless" is a tiny
fraction of cloud use let alone overall use.

~~~
Touche
Some times hype is warranted. Get passed it and look at the actual technology
and its value. If you are building a website in 2018 you are doing yourself a
disservice manually managing servers.

~~~
qaq
at our scale if we used "serverless" our AWS bill would go from several mil a
month to 20+ mil. plus

~~~
Touche
You're not a startup.

~~~
qaq
Isn't a goal of startup to grow? You lock yourself into a model where you
compute spend is 10X that of your competitors you not gonna get very far.

~~~
Touche
At a certain scale using cloud services at all is too expensive. At a certain
scale you write your own databases. You make your own chips.

