
A fast HTTP request/response parser for Common Lisp - fukamachi
https://github.com/fukamachi/fast-http
======
eudox
For context, Eitaro Fukamachi has been working on a whole bunch of things
relating to the low-level parts of Common Lisp web development: A fast HTTP
parser, a fast URI parser[0], a libuv-based web server[1] (That outperforms
Node!), and he's also written Clack[2], which is Common Lisp's Rack/WSGI.

[0]: [https://github.com/fukamachi/quri](https://github.com/fukamachi/quri)

[1]: [https://github.com/fukamachi/woo](https://github.com/fukamachi/woo)

[2]: [https://github.com/fukamachi/clack](https://github.com/fukamachi/clack)

~~~
rgbrgb
> (That outperforms Node!)

In what way? All of the benchmarks I'm seeing on that page indicate otherwise.

~~~
eudox
When he first posted it, it did, it looks like recent changes have brought
that down:

[https://github.com/fukamachi/woo/commit/be4a0db688af05ee204e...](https://github.com/fukamachi/woo/commit/be4a0db688af05ee204ec37c5c185f13a4fe8e36)

~~~
fukamachi
Right. When I added default headers, the score went way down.

Probably libevent2 is the bottleneck and no way to make it faster without
switching its backend (the cl-async's author is working on rewriting it with
libuv now).

~~~
aidenn0
Out of curiosity did you try basic-binary-ipc? I would expect it would lose
out to cl-async, but have never benchmarked the two against each other.

------
TheMagicHorsey
I've always assumed Clojure was the right choice for a Lisp to do web
applications. Perhaps I should revaluate my assumptions.

~~~
apgwoz
If you're assuming this based on some microbenchmarks, you should probably
instead reevaluate your evaluation techniques.

(This isn't meant as a personal jab, but too many people focus on silly things
like microbenchmarks when choosing tools, and _not_ on other more important
things like support avenues, tooling, libraries, etc.)

~~~
aroman
I don't think it's fair to assume that TheMagicHorsey was basing his
evaluation on the microbenchmarks themselves.

The real story here is that people are working on a number of new
libraries/modules/tools for doing web development in CL — the benchmarks (as
you correctly point out) are a minor detail.

~~~
apgwoz
Perhaps you should read more carefully. "If you're assuming ..."

It's just _not_ clear from the context we have. But, caution! Microbenchmarks
should only be Microtrusted.

------
thu
In the HTTP benchmark game, there is also Bryan O'Sullivan's attoparsec-based
parser:
[http://www.serpentine.com/blog/2014/05/31/attoparsec/](http://www.serpentine.com/blog/2014/05/31/attoparsec/)
The important point is that the parser is a few dozen lines of code (i.e. it
relies on a general parsing library instead of being custom hand-written
code).

------
Animats
Is parsing HTTP headers a major bottleneck? HTML parsing, sure, especially
with the horrors of HTML 5 error handling per the spec and the need to back up
after guessing the character set. But HTTP headers are not that big or
complex.

~~~
eudox
From the Reddit thread:
[http://www.reddit.com/r/lisp/comments/2klhpn/quri_a_uri_libr...](http://www.reddit.com/r/lisp/comments/2klhpn/quri_a_uri_library_for_common_lisp_6x_faster_than/clmsfzd)

>Right. When I was profiling Wookie, I found URL parsing was the third
bottleneck (the first is HTTP request parsing, and the second is libevent2).

~~~
Animats
Yes, that's really important for compatibility with Google Wave.

------
verytrivial
This reminds me of John Fremlin's work back in 2009
[http://john.freml.in/teepeedee2-c10k](http://john.freml.in/teepeedee2-c10k) .
(Hello, John!)

------
wspeirs
Just curious what about this parser makes it faster? Can anyone give a basic
explanation?

~~~
easytiger
Well if you implemented the c parser with precisely the same algorithms the c
one would become faster, so one can assume they have totoally different
underlying implementations

~~~
haberman
That is not necessarily true (and I say this as someone who heavily favors C
for parsers). An implementation with a JIT will likely be able to optimize
away work by noticing that the table of callbacks is empty.

I would be more impressed if the callback table was not empty (and the
callbacks actually did something, like increment an integer) and Lisp was
still winning.

EDIT: I see others saying that SBCL is AOT, not JIT. If so I'd be very
interested to look at the disassembly side-by-side and see an explanation for
why SBCL is winning.

~~~
muyuu
SBCL is a compiled Common LISP (even in interpreted mode, it compiles every
eval-ed lambda before calling it).

On the other hand there is nothing inherently stopping the optimiser from
optimising the resulting code as well or better than GCC or any C compiler.
Theoretically it could produce optimal machine code.

~~~
haberman
Correct me if I'm wrong but I was under the impression that Common Lisp is
memory-safe (implying guards like bounds-checks), dynamically typed (implying
type checks) and garbage collected (implying GC overhead).

~~~
muyuu
Yep but all these checks don't necessarily need to happen at runtime. A clever
enough compiler can optimise all of it (or most of it) at compile time. In
general this is a very hard problem but for something static enough (like a
simple parser) the compiler can produce code and determine its safety without
needing to carry checks on to the runtime.

Note that C code or machine code would also have to perform input checks at
runtime when doing http header parsing. A Common Lisp compiler wouldn't
theoretically need to produce code performing any more runtime checks than
these.

~~~
_delirium
You can also manually turn some checks off. In some Lisp systems, declaring an
optimization setting of (safety 0) will turn off a bunch of checks, even ones
that the compiler can't statically determine are safe to skip. Of course this
should be used very carefully if at all (especially in code that's parsing
arbitrary data from the internet!).

------
halayli
parsed = http_parser_execute(parser, &settings_null, buf, strlen(buf));

assert(parsed == strlen(buf));

He's calling strlen(buf) twice here. There's no guarantee that this is
optimized at compile time. Usually len of buf is determined from read/recv
system calls and for a fair comparison strlen should be removed here.

~~~
fukamachi
No difference even though I ran the benchmark again after removing the line.

~~~
halayli
Interesting. I was expecting at least cpu branch improvements given the lack
of assert.

------
serbrech
Hmm, I can't really write lisp, but, how maintainable is this??
[https://github.com/fukamachi/fast-
http/blob/master/src/parse...](https://github.com/fukamachi/fast-
http/blob/master/src/parser.lisp)

~~~
adwf
It could do with some comments and doc-strings certainly, but I'd give it a
pass for now as it's a new project. As for the code, I imagine that in any
language, highly optimised programs can start looking pretty odd after a
while. Try understanding "grep" for example:

[http://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c](http://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c)

As a guy not particularly proficient in C, that looks crazy to me. Although it
does at least have decent comments, which I'd expect for such a mature
project!

------
haberman
I was not able to run the benchmark for Lisp. What Lisp implementation is
expected? I tried CLISP and got:

    
    
        *** - READ from #<INPUT BUFFERED FILE-STREAM CHARACTER #P"benchmark.lisp" @1>: there is no package with name "SYNTAX"

~~~
eudox
I'd recommend SBCL, since a high-performance JIT compiler is likely to give
more representative results than a bytecode VM like CLISP. Plus, I'd expect
some minor portability errors to arise when you push the envelope like this.

By the way, your error seems to be that the cl-syntax library is not being
found, as opposed to a portability problem, so maybe you don't have Quicklisp
installed and loaded?

~~~
gjm11
Pedantic correction: Unless something's changed very radically since I last
looked, SBCL's compiler is AOT not JIT. (Though in interactive use it compiles
things as you enter them, which I suppose is kinda JIT if you look at it from
just the right angle and squint.)

~~~
eudox
You're right, and I always make this mistake for the reason that AOT/JIT look
the same from the outside.

------
_RPM
Is Lisp a write-only language?

~~~
adwf
I'll assume this is an honest question - not a troll (and hence don't deserve
the downvotes), but no, Lisp isn't write-only.

It's an immensely flexible language that lets you code in a variety of
different styles, so you _can_ end up with code spaghetti. But that doesn't
mean that you have to, or even that it's the most likely thing to happen.

If you compare it to idiomatic Perl, where a lot of the community seem to get
obsessed with the famous one-liners, Lisp seems a lot better to me. And if you
follow a decent style guide like:

[https://google-
styleguide.googlecode.com/svn/trunk/lispguide...](https://google-
styleguide.googlecode.com/svn/trunk/lispguide.xml)

Then there's no particular reason why you should end up with disaster code
anymore than in any another language. I'd rather read a single page of Lisp,
than the equivalent numerous pages of Java or some other verbose language.

