
Which implementation of Lisp is best when speed matters? - palish

======
palish
Cool, thanks for all the great replies. Here's a little about what I'm doing.

I've worked in the video game industry for the last two years, and I've been
messing around with Direct3D/OpenGL for about 5. I believe wrapping these
API's in an extremely high level language like Lisp might speed up creation of
a video game by about an order of magnitude, especially if I expose only very
high level graphical concepts to Lisp, instead of just exposing every API
function.

Speed is crucial. I've layered Ruby onto my engine, but the problem was that
if I wanted 60 frames per second, I could only do 16,000 Ruby if-statements
per frame. That wouldn't work at all.

Even if the high level language I choose is only 4-10 times slower than C++,
that would be fine. Ruby was about 100-200 times slower.

Investigating Lisp, I only require two things:

1) Somehow, I expose C functions to Lisp. These functions take standard types
as arguments (float, int, double, char ...).

For example, I might expose, from C:

(Edit: argh, I can't write a star on News.YC. The following should read:
"const char (star)str")

void print_to_game_console(const char str);

Then in lisp, I'd then be able to do (print_to_game_console "Odelay!")

Something along those lines.

2) Lisp only has access to what I expose. I won't get into the boring details,
but I'm distributing the Lisp script files for anyone to edit and run on
other's machines, so sandboxing Lisp somehow is also pretty crucial. As long
as under no circumstances can Lisp access the file system, then everything
else is fine.

Thanks everyone! Shawn

~~~
gyro_robo
Also see SWIG: <http://www.swig.org/Doc1.3/Lisp.html>

Here's an example of compiling and loading inline C code that works with SBCL
and CMUCL: <http://homepages.nyu.edu/~ys453/>

SBCL: sb-alien package allows interfacing with C-code, loading shared object
files, etc. See Foreign Function Interface.
<http://www.sbcl.org/manual/Foreign-Function-Interface.html>

~~~
Todd
I read about Google's use of Python here

<http://panela.blog-city.com/python_at_google_greg_stein__sdforum.htm>

a few weeks ago and saw the part where they mentioned SWIG. This was quite an
insight for me since I've been trying to figure out for a years the best way
to write binary extensions for Perl, Lisp, etc. To see that Google uses it as
a matter of course completely sealed it for me.

I have integrated it into my project over the last couple of weeks and I've
got to say it is the bee's knees. It took some thrashing to get started and
there are still some dark corners (e.g., typemap voodo) but I will never go
back. My first project is a high performance sockets IPC module patterned
after memcached/libevent and it works like a charm.

I highly recommend SWIG.

------
papersmith
The ones that compile to native code:

\- Lispworks (commercial)

\- Allegro CL (commercial)

\- SBCL (my pick)

\- CMUCL

\- OpenMCL (Mac only)

\- GNU CL (not standard-comforming, but works well with other GCC binaries)

\- Corman CL (commercial, windows only)

This is just off of the top of my head, please add anything I've missed.

Assuming you're doing web apps:

<http://groups.google.com/group/comp.lang.lisp/msg/352f440baa7aeb94>

They (cl-user.net) have pretty impressive results with a single Lispworks
process in a production environment; ~3.3 requests/sec, ~5million
requests/day, uptime 456 days.

It seems that historically when you want performance+portability+paid-support
you go for one of the commercial implementations. CMUCL used to be the choice
among free implementations for performance. SBCL inherits the performance of
CMUCL, and they reworte some of the layers to be more portable. They had some
threading issues in FreeBSD, but they're now resolved and pretty solid after
recently passing version 1.0. SBCL now has a pretty substantial user base, so
you should be able to get help by just asking around.

~~~
omouse
And now to choose which one to use! Fun! :P

------
gibsonf1
Here's a discussion about the various Lisp options:
<http://programming.reddit.com/info/1edrw/comments>

We use Lispworks, but if there is a better (especially freer) option, we would
switch in a heartbeat. (We don't use Allegro Cache - so that wouldn't hold us
back from switching). So far our systems are running very fast with LW - no
complaints on speed.

------
vikram
SBCL on linux. Lightening fast. In some benchmarks it even beat C. The thing
to remember with lisp is that you'll need to work to get your program to be
efficient. It's not too much work but it's definitely something you'll need to
do.

~~~
palish
Thanks, I'll remember! It seems like SBCL is the general consensus. I hope I
can figure out a way to get it to interface with a C library.

------
mattculbreth
Well I know that Paul has Arc running on MzScheme, and I've found this site to
be plenty fast. Not exactly what you asked, but if you haven't started coding
you could consider that.

~~~
dfranke
I've had bad experiences with PLT. I used it for my senior project and ended
up having to rewrite in Common Lisp because I kept running into bugs in the
interpreter. I'm guessing that Arc compiles down to a pretty well-tested
subset of PLT, but if you start messing around with obscure libraries or
really beat up hard on the macro system, you're asking for trouble.

------
dfranke
Probably SBCL.

~~~
busy_beaver
If Scheme will do, Chicken, Bigloo, and Gambit all compile to C code as an
intermediate step (which you then run through gcc or whatever to generate the
binary). Bigloo will also generate JVM and .NET output (.NET is still
experimental).

I haven't worked much with Gambit, but have fooled around with both Chicken
and Bigloo. Of the two, I found Chicken to be the friendliest (it also seems
to have a somewhat more active community). Another possible advantage of
Chicken is that it's directly supported by SWIG. There was some activity on
the Chicken mailing list a few months back about porting some game libraries.
I don't know what the current status of that might be.

------
marketer
The question should be: if you want speed, is it really worth using lisp?
Maybe your time would be better spent writing a c module and then interfacing
with the lisp interpreter.

I'm no lisp hater, it's a great language that offers some neat abstractions.
But to really optimize code, you have to be intimately familiar with how data
structures are implemented within the interpreter. You can avoid this problem
by using C.

~~~
herdrick
Actually you're likely to write slower code if you start with C, instead of
starting with a high level language and then _after finding out what is too
slow_ , writing those parts in C.

Except it won't be too slow. Especially if you use one of the popular
implementations of Common Lisp or Scheme. They're pretty fast.

More to the point, you're killing yourself on productivity if you are
approaching things from a performance-central perspective. My advice to you is
to forget C.

~~~
marketer
It's sometimes difficult to fix performance issues in your app unless you know
what's causing the slow-down in the interpreter, but that's usually a pain.
Lisp is a constructive language, so a lot of time is spent doing memory
allocation. Interpreters often optimize that by pre-allocation and such, but
to diagnose that you need to know exactly when memory is allocated, etc..

AN example: I was using the python heap module to implement a priority queue
with several hundred thousand values. The memory allocation caused by the
dynamic growth of the list was crippling the performance. I re-implemented it
as a binary heap in C, and it was asymptotically faster.

~~~
shiro
You're half right. Tuned Lisp code can be as fast as C, but you have to know
the compiler so intimately that you can tell what kind of machine insturctions
it is generating; you'll use "disassemble" a lot ("disassemble" is a part of
CommonLisp standard---you may get a sense how Lispers are performance freak).
Usually the bottleneck part is tuned to the point that it won't do any
allocation and run-time type dispatch at all.

There's one big advantage of using Lisp over C for performance: Macros. During
optimization it is typical that you have to write several versions of code,
changing bits and pieces, and run benchmarks to see what is optimal. Macros
allow you to generate different versions of code from simple changes of
parameters, without incurring overhead of function call/variable reference
etc. (You can do similar thing in C++ templates, but Lisp macros allows much
more). If you can easily parameterize your code, you can try more ideas and
run more benchmarks, so it is more likely that you'll find better
optimization.

AN example: Allegro CL version 7 and later has Perl-compatible regular
expression library, purely written in Lisp, that runs faster than Perl (at
least at the moment I wrote it, using benchmark suite came with PCRE).

------
chris_l
What are you doing and which part is the speed bottleneck?

As I've mentioned before somewhere on here, I'm building a blog search engine
in Common Lisp. That is indeed performance critical as you can probably
imagine and I've found SBCL to work very well.

------
hayeah2
use assembling language then wait for Greenspun's 10th.

------
albertcardona
When speed matters, what you need is a good hacker to make a smart-ass elegant
program that goes around the bottle-necks. But I'll attempt not to avoid your
question altogether: from my own personal testing, SBCL is fastest. My bias is
my own benchmarks. Find your own set.

