
Hotswapping Haskell - trevorriles
http://simonmar.github.io/posts/2017-10-17-hotswapping-haskell.html
======
teraflop
I don't want to knock the technical achievement here -- it's a cool hack --
but I'm really surprised that it was deemed to be the best choice for a
production system.

In the first place, "we can't compile our code on every change because it
takes too long" is a really awful situation to be in. Are developers not
building and testing their changes before deploying them? Can Facebook not
afford a continuous integration system that can run builds in parallel? It
sounds like this problem is only happening because the application is a giant
monolith, but for some reason splitting it up would slow down development even
more... I'm not sure I buy that reasoning.

The article says that "Haskell’s strict type system means we’re able to
confidently push new code knowing that we can’t crash the server", which is a
real stretch. In addition to all of the usual ways a computation can diverge,
this hot-swapping system adds a whole new variety of failure modes. The
article talks about how the code needs to be carefully audited to prevent
memory leaks, but it doesn't even mention the weird things that can happen
when mutable state is preserved across code modifications. Debugging is a pain
when your data structures can get into states that aren't reachable with any
single version of the code. (This is a well-known issue in Linux kernel live-
patching, for instance.)

~~~
jimbokun
"In the first place, "we can't compile our code on every change because it
takes too long" is a really awful situation to be in."

Isn't this exactly the problem Go was invented to solve?

~~~
kornish
It was one of them. However, given the other writing/talks Facebook has put
out about their usage of Haskell and Haxl, Go is probably not a good fit for
their use case due to language expressivity concerns (not declarative enough,
not enough type safety, not syntactically flexible enough for writing DSLs).

------
rdtsc
Hotswapping just like security is one of those things that is hard to bolt on
later, unless it is built in deeply into the very core of the language /
runtime.

Erlang (and Elixir) define hotswapping very well. It is a standard way to
upgrade code in production in some places. And even with it being well defined
it is still very hard and there are enough corner cases to handle.

But when used correctly, it is really magical and can achieve nice properties.

Besides just upgrading code, hotswapping (at least in Erlang) can be used for
debugging -- you can update the running code with extra log statements to
catch sneaky corner cases. Maybe it is a customer setup, that is very hard to
replicate.

Or you can use it for local development, as you edit code, the module gets
auto-reloaded (with a helper).

It can also be used to deliver hot fixes. Say if the fix is simple and the
customer cannot wait for a full release to be built, can update their system
on the spot to tie them over. Not idea but I've seen it save the day many
times.

~~~
crusso
_Or you can use it for local development, as you edit code_

This is a huge feature for me in my Elixir development. I mostly use Elixir
for some server code that manages many connections to external network
entities. It would be a huge hassle to bring down my server application every
time I want to make a change.

With Elixir (yeah, Erlang), I can normally recompile the module I'm working on
and deploy it in the running server. Not only is it a good way to constantly
observe Erlang hot-swapping in action on my dev machine, it's a huge time
saver.

------
tekacs
People might find this interesting to compare and constrast to (certainly I
noticed parallels with[1]) Netflix's 'serverless' platform, discussed
yesterday:

[https://medium.com/netflix-techblog/developer-experience-
les...](https://medium.com/netflix-techblog/developer-experience-lessons-
operating-a-serverless-like-platform-at-netflix-part-ii-63a376c28228)

Edit: As a sibling commenter notes this is most eminently doable with e.g.
Common Lisp and BEAM (Erlang/Elixir), but more folks are (publicly) attempting
this in other environments now (I've experimented with a number of approaches
to this the last few years, so I'm trying to keep score - would love to see
any comments on other attempts below).

[1]: Quote: "At the core of the redesign is a Dynamic Scripting Platform which
provides us the ability to inject code into a running Java application at any
time. This means we can alter the behavior of the application without a full
scale deployment."

~~~
cies
> Common Lisp and BEAM (Erlang/Elixir)

Clojure as well right?

~~~
tekacs
Yup having strong namespaces in general makes this easier I think and in
Clojure tools like devcards (CLJS) and Ring reload (CLJ) support this
workflow.

------
aaron-lebo
It's basically this:

[https://news.ycombinator.com/item?id=8804381](https://news.ycombinator.com/item?id=8804381)

[http://nullprogram.com/blog/2014/12/23/](http://nullprogram.com/blog/2014/12/23/)

You'd be surprised how many languages can do this. Though it's hard to beat
lisp (and Erlang), where it is the default.

~~~
lallysingh
really, anyone who bothers reading the dlopen(3) manpage:
[http://man7.org/linux/man-
pages/man3/dlopen.3.html](http://man7.org/linux/man-pages/man3/dlopen.3.html)

Loading / unloading code is straightforward. The trick is in getting the code
called from existing code.

~~~
pjmlp
Loading is straightforward, unloading depends.

------
chillee
Not really relevant to the article itself, but I've always been surprised how
much attention Facebook's Haskell usage gets compared to Facebook's Ocaml
usage.

From interning at Facebook, the only project that I'm aware of that uses
Haskell is Sigma. On the other hand, numerous projects use Ocaml: Infer, HHVM,
Flow, ReasonML, Pfff, etc.

However, it's Haskell that gets all the attention on Hacker News.

~~~
MaxGabriel
I’ve seen them recruit for a second Haskell team, I believe doing internal
analytics or machine learning.

------
GarvielLoken
We are going to be back in the 70ies/80ies of running live image programming-
systems in no time. History repeats itself. Mark my words.

~~~
pjmlp
That is what web apps are all about actually.

