
Running Lisp in Production - f00biebletch
http://tech.grammarly.com/blog/posts/Running-Lisp-in-Production.html
======
white-flame
We deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A
few specifics that I'd point out:

Many of SBCL's optimizations are fine grained selectable, using internal SB-*
declarations. I know I was at least able to turn off all optimizations for
debug/disasm clarity, while specifically enabling tail recursion so that our
main loop wouldn't blow up the stack in that build configuration. These aren't
in the main documentation; I asked in the #sbcl IRC channel on FreeNode.

You can directly set the size of the nursery with sb-ext:bytes-consed-between-
gcs, as opposed to overprovisioning the heap to influence the nursery size.
While we've run in the 8-24GB heap ranges depending on deployment, a minimum
nursery size of 1GB seems to give us the best performance as well. We're
looking at much larger heap sizes now, so who knows what will work best.

While we haven't hit heap exhaustion conditions during compilation, we did hit
multi-minute compilation lags for large macros (18,000 LoC from a first-level
expansion). That was a reported performance bug in SBCL and has been fixed a
while back. Since the Debian upstream for SBCL lags the official releases
quite a bit, it's always a manual job to fetch the latest versions, but quite
worth it.

Great read, and really familiar. :-)

~~~
vseloved
We played with sb-ext:bytes-consed-between-gcs, but couldn't find the right
balance. That's why we were surprised with the result of the oversized heap
experiment

~~~
white-flame
You might want to do some googling about it, but I also seem to remember that
SBCL might decide to use large or small MMU page granularity depending on the
size of the heap. That might be the watershed trigger for performance. (or
just some random mis-remembered nonsense)

------
Grue3
Common Lisp's macros and grammar go together like bread and butter. A grammar
module in the app I built [1] uses macros to generate huge amounts of
repetitive code.

[1] [https://github.com/tshatrov/ichiran/blob/master/dict-
grammar...](https://github.com/tshatrov/ichiran/blob/master/dict-grammar.lisp)

I wonder if they're still hiring Lispers. I once passed on the opportunity to
work in their Kiev office, but I might give it a shot again.

~~~
muraiki
Wow, I checked out your ichi.moe app. It's awesome! I'm definitely going to
use this as I work on re-learning the Japanese I've forgotten over the years.
I've done some Racket and Clojure but never done CL... I'll have to check it
out.

~~~
brobinson
Thanks for pointing this out. I'm learning Japanese now and this will be
massively helpful.

------
orthecreedence
Great article, and good reminder on using trace. Every time I rediscover
trace, I can't remember how I ever forgot to use it in the first place for
most of my problems.

I used CL in a production environment a while back for a threaded queue worker
and nowadays as the app server for my turtl project, and I still have yet to
run into problems. It seems like you guys managed to push the boundaries and
find workable solutions, which is really great.

Thanks for the writeup!

------
doomrobo
Slightly off-topic, but does anybody know of a kind of "LISP Challange" set? I
recently started the Matasano challenges[0] and I found them really well-
suited to my style of learning (learning by doing and expanding by reading
relevant material, enabled by my own internal motivation). Is there anything
like that that has a relatively small set of condensed yet rich challenges
that demonstrate key elements from LISPy functional programming? I read some
of SICP but reading long form really puts a damper on my
motivation/excitement. Also there were a lot of exercises (with a lot of
overlap in concepts) so I didn't know what to do and what not to do, since I
wasn't about to do every single one. Any pointers would be much appreciated!

[0] [http://cryptopals.com](http://cryptopals.com)

~~~
Syssiphus
From the top of my head..:

The Praetorian Bootcamp challenges are similar to the Matasano ones:
[https://www.praetorian.com/challenges/](https://www.praetorian.com/challenges/)

Lisp Koans: [https://github.com/google/lisp-
koans](https://github.com/google/lisp-koans)

Project Euler: [https://projecteuler.net/](https://projecteuler.net/)

Exercism: [http://exercism.io/](http://exercism.io/)

99 Lisp Problems:
[http://www.ic.unicamp.br/~meidanis/courses/mc336/2006s2/func...](http://www.ic.unicamp.br/~meidanis/courses/mc336/2006s2/funcional/L-99_Ninety-
Nine_Lisp_Problems.html)

~~~
monkeyshelli
I've been using python-koans to teach Python, but the thought of looking for
lisp-koans from Github didn't even cross my mind now as I personally started
learning CL. Thanks for these excellent links!

------
davexunit
Awesome stuff. Articles like this are what we Lispers/Schemers need to show
that our languages can be used for "real work"(tm).

------
PuercoPop
One of the things I would have liked to see on the article is how do they
handle the deployment itself. Do they build an executable with build app? To
they used sb-daemon? An home-grown solution using sb-posix:fork?

~~~
lisper
I don't know how they do it, but I have run several Lisp production systems. I
use CCL, which has a really fast compiler. So I just load the code from
source. The deployment process then becomes: git pull, quit and restart Lisp.
(The ccl-init file loads the code.)

~~~
PuercoPop
And in that case how do you handle the monitoring of the system?

~~~
lisper
I don't understand the question. Once it's running, it's like any other
server, and you monitor it like you would any other server written in any
other language.

~~~
WalterGR
Speculation here, but based on the context, I think the question is: Some
daemon-izer solutions will monitor the daemon and e.g. restart it if it's
unresponsive. How do you handle this slash how is this handled in the Common
Lisp world?

~~~
lisper
As a last resort, if Lisp becomes totally unresponsive, you kill the process
and restart, same as any other language. But it's pretty rare to lose the
REPL, so usually you can fix any unexpected problems through that.

------
dfischer
Is it worthwhile to explore Clojure for web-dev seriously or more as a toy?

~~~
bitsai
I vote that it's worthwhile as well. We've been using Clojure full-time at
ReadyForZero for over 3 years now (for web-dev as well as other tasks), and
are very happy with our choice.

~~~
copperx
Is it better than plain Java? I mean, if you have the entire Java ecosystem,
is it worth it to go upstream just to work in a Lisp-like language?

~~~
bitsai
IMHO, absolutely. There are many wins, but the 2 biggest for us have been:

1\. Clojure's tools for making abstractions are orders of magnitude better
than Java's, so our codebase is significantly smaller and simpler than what
it'd look like in Java.

2\. Clojure lends itself very well to a REPL-oriented kind of iterative
development that speeds up writing and testing code tremendously, compared to
the typical Java workflow.

------
jon-wood
Apparently they use "JVM languages", JavaScript, Python, Go, Lisp and Erlang
in production.

I may be in the minority, but that would drive me mad. I assume they're not
routinely jumping between those stacks multiple times a day, but even so is
there really that much benefit that it's worth keeping track of how to do
things in that many different environments?

~~~
jessaustin
Since "proper service encapsulation" is mentioned, it may be that each team
uses whatever they like, and as long as your component speaks http you don't
have to look at what other components are doing.

~~~
krvss
This is exactly how it is (except sometimes it is not HTTP but message queuing
etc)

------
BlanketLogic
Very informative. Thank you.

Anyone here has any experience with the GCs of Allegro or LispWorks or any
other commercial Lisp implementations?

~~~
lispm
LispWorks has a lot of features in its GC. Years ago it was used in a
demanding telco application, an ATM switch. It was also used on a space
mission experiment. Generally the runtime is very very nice.

Franz Inc uses Allegro CL in a large database. They tuned the GC quite a bit
for that. But there were also other GC demanding applications on Allegro CL,
for example in CAD and 3D design. They are now working on a concurrent GC,
something which is still rare in the Lisp world.

~~~
lambdaelite
Why are concurrent GCs rare?

~~~
vseloved
are there many platforms, besides JVM and .Net, that have good-quality
concurrent GCs?

~~~
fredyr
The Erlang Vm (BEAM), is another one at least.

~~~
the_why_of_y
It doesn't actually implement concurrent GC, although what it does implement
is far simpler and has a similar effect (low latencies) as concurrent GC.

Each Erlang process has a separate heap that is collected independently;
because the process heap is usually small a stop-the-process collection does
not take much time.

The downside is that sending messages between processes requires copying all
the data that is sent between process heaps.

------
outworlder
Aw, now they have disclosed their secret weapon! [1]

[[http://www.paulgraham.com/avg.html](http://www.paulgraham.com/avg.html)]

------
akssri
> but we value choice and freedom over rules and processes.

Which I exactly why I feel Lisp doesn't see much use elsewhere :(

------
lkrubner
Good lord, I would go insane if I ran into a bug like this:

"We've built an esoteric application (even by Lisp standards), and in the
process have hit some limits of our platform. One unexpected thing was heap
exhaustion during compilation. We rely heavily on macros, and some of the
largest ones expand into thousands of lines of low-level code. It turned out
that SBCL compiler implements a lot of optimizations that allow us to enjoy
quite fast generated code, but some of which require exponential time and
memory resources. "

~~~
apalmer
I dont really understand the downvotes... I agree with you would go nuts.

My two cents, if your macro is expanding to thousands of lines of code, your
doing something wrong I think. I would expect Macros to expand out to a few
lines of code which might have function calls that themselves may contain
however many lines of code... but expanding to thousands of lines INLINE via
macros seems wrong.

~~~
jeremiep
Some macros expand to a _lot_ of code, especially when doing more in-depth
transformations such as those performed by core.async in Clojure where they
transform standard sequential code into a state machine with exactly the same
semantic but with the added ability to execute, yield and resume like a
coroutine.

------
welshguy
I love it :0) I tried grammarly, and typed in a remembered poem. It informed
me that it had detected significant plagiarism.

Edit: It's still not advice I would pay for, though.

------
avodonosov
Thanks, great post and lot of useful references.

------
zenogais
This was fantastic. Thank you.

------
mud_dauber
Wow. I can't ever remember reading about a consumer-facing app using Common
Lisp. Ever.

~~~
jon-wood
You're using one ;)

~~~
cpach
Nope, HN is written in Arc (which runs on top of Racket) :-p

------
jjawssd
Once you go deep enough you are fucked no matter what language you choose.
Might as well pick one that doesn't beat you up too much.

------
eruditely
Why not racket?

~~~
dkvasnicka
I was thinking the same question and the reply would probably be
"performance". I don't think untyped Racket can currently compete with SBCL
and I'm not sure they'd be interested in Typed Racket. But I'd love to be
proven wrong on both of those things ;)

~~~
eruditely
Ok that makes sense.

~~~
vseloved
But that's not the reason. Personally I find Lisp superior in most respects,
but I don't think it's a good place and time to argue on Lisp vs Racket and
similar topics.

