
Why Lisp? - lisper
http://blog.rongarret.info/2015/05/why-lisp.html
======
WalterGR
The article doesn't discuss macros, which is one of the answers to "Why Lisp?"

I didn't "get" macros until I read a footnote in the (freely available) book
_Practical Common Lisp_. In chapter 7, it introduces the `dolist` macro.

    
    
        DOLIST loops across the items of a list, executing the loop
        body with a variable holding the successive items of the list.
        This is the basic skeleton (leaving out some of the more
        esoteric options):
    
        (dolist (var list-form)
          body-form*)
    
        ...For instance:
    
        CL-USER> (dolist (x '(1 2 3))
                   (print x))
        1
        2
        3
        NIL
    

Buried in a footnote is this:

"DOLIST is similar to Perl's `foreach` or Python's `for`. Java added a similar
kind of loop construct with the 'enhanced' for loop in Java 1.5, as part of
JSR-201.

Notice what a difference macros make. A Lisp programmer who notices a common
pattern in their code can write a macro to give themselves a source-level
abstraction of that pattern. A Java programmer who notices the same pattern
has to convince Sun that this particular abstraction is worth adding to the
language. Then Sun has to publish a JSR and convene an industry-wide "expert
group" to hash everything out. That process--according to Sun--takes an
average of 18 months. After that, the compiler writers all have to go upgrade
their compilers to support the new feature. And even once the Java
programmer's favorite compiler supports the new version of Java, they probably
still can't use the new feature until they're allowed to break source
compatibility with older versions of Java.

So an annoyance that Common Lisp programmers can resolve for themselves within
five minutes plagues Java programmers for years."

[http://www.gigamonkeys.com/book/macros-standard-control-
cons...](http://www.gigamonkeys.com/book/macros-standard-control-
constructs.html)

~~~
Xophmeister
Could someone please explain the difference between Lisp macros and, say,
languages that have first-class functions? I get that a Lisp macro will be
expanded into the respective code, while a function's execution is different.
However, at the practical (i.e., developer's) level, are there any additional
benefits?

Can, say, a Lisp macro be 'partially formed', in the sense that it can expand
into some boilerplate that represents an incomplete syntax tree? (Whereas a
higher-order-function is necessarily complete.) I can see that being useful,
but not inasmuch a it's made out.

~~~
snikeris
My favorite example for this is the lame idiom you see in Java code:

    
    
        if (log.isDebugEnabled()) {
            log.debug("expensive" + debug + message);
        }
    

This is "better" than just log.debug(...) because with the latter, your
expensive log message argument needs to be evaluated even if debug is
disabled.

However, in a language w/ macros, you just say:

    
    
        (debug (str "expensive" debug message))
    

and these considerations are already taken care of for you:

[https://github.com/clojure/tools.logging/blob/master/src/mai...](https://github.com/clojure/tools.logging/blob/master/src/main/clojure/clojure/tools/logging.clj#L91)

~~~
kedean
For completeness of the argument, this particular problem is solved in the
Java world with string formats. With the slf4j interface, that would be:

log.debug("expensive %s %s", debug, message)

The message is not actually formatted into one string unless the DEBUG trace
level is enabled. Of course, you are still passing the arguments around, but
with object references that's a negligable difference.

I still appreciate the solid example of a problem macros are good at solving,
though. Two ways around one problem.

~~~
nulltype
What if it's debug() instead of debug? That could be an expensive function
that does a bunch of things to produce the debug output.

~~~
kedean
Solid point, that would be a downside of the slf4j approach.

------
orthecreedence
The interactive model is insanely cool. When building a toy game engine a
while back
([https://github.com/orthecreedence/ghostie](https://github.com/orthecreedence/ghostie))
I saved probably half the development time by being able to redefine
functions/values _while the game was running_.

The old way of lisping is to prototype in lisp, then build in a "real"
language (c/java). However nowadays the lisp implementations (CCL/SBCL
specifically) are fast/advanced enough to the point where you can prototype in
lisp, then just use add some type specifiers and boom there's your app. Hell,
with ECL you can even embed your lisp program into another one, while still
achieving compiled speeds.

~~~
demilicious
What sort of tools are required for this? I'd like to start taking Lisp
seriously, but workflow stories like this tend to hinge on using Emacs/SLIME.
Is that always the case?

EDIT: I suppose what I'm asking is whether you could elaborate more on what
this looks like, in practice.

~~~
orthecreedence
Actually no, I use vim/slimv almost exclusively. I think Sublime2 might also
have features that let you "hook into" a remote REPL, but I'm not very
familiar. There may be other editors with lisp integration as well, but I'm
not sure what they are.

If you do like vim, slimv is a really great option for lisping.

~~~
laichzeit0
Does slimv work with Racket yet? I remember trying this a few months ago and
giving up.

------
munificent
> The reason that code represented as XML or JSON looks horrible is not
> because representing code as data is a bad idea, but because XML and JSON
> are badly designed serialization formats.

By that same token, a Volkswagen Beetle is a badly-designed boat.

XML was never designed as a data serialization format. It's a _markup
language_. It was designed to sprinkle structure and metadata into large
human-readable plaintext documents.

Likewise, JSON is a subset of a general-purpose programming language's literal
notation that happened to be very fast to parse in a browser by virtue of the
browser implementing that language.

Personally, I don't think s-exprs are a particularly great serialization
format either. The problem is that there's no one-sized-fits-all for
serialization. What we value is brevity, but basic information theory tells we
can only make expressing some things more terse by making others more verbose.

When you say some format is badly-designed, all you're really saying is that
it isn't optimized for the kinds of data you happen to want to serialize.

~~~
lisper
> XML was never designed as a data serialization format. It's a markup
> language.

Those two things are not mutually exclusive.

> Likewise, JSON is a subset of a general-purpose programming language's
> literal notation that happened to be very fast to parse in a browser by
> virtue of the browser implementing that language.

That's true. That is not in conflict with anything I said.

> The problem is that there's no one-sized-fits-all for serialization.

No, that's not true. S-exprs really are a global optimum in the space of
serialization designs. All the alternatives are logically equivalent to
S-exprs but with extra punctuation that makes them arguably harder to read,
but inarguably harder to write. That is why S-exprs are the ONLY syntax ever
designed (some would say "discovered") by humans that has been successfully
used to represent both code and data.

~~~
CJefferson
Is there any loss less binding of XML to s expressions? I've never seen one.

Usually example where I see people rewrite XML to s expressions (like in this
thread) are very lossy -- its easy to be pretty by throwing away most of the
information!

~~~
lisper
> Is there any loss less binding of XML to s expressions? I've never seen one.

You haven't looked hard enough. Mappging XML to sexprs is trivial:

<tag value=attr ...>content</tag>

\-->

((tag value attr ...) content)

~~~
CJefferson
True, but to be honest, that isn't really any more useful compact (well, it
saves closing tags I suppose).

------
dang
> Among other things, it makes writing interpreters and compilers really easy,
> and so inventing new languages and writing interpreters and compilers for
> them becomes [...] a part of day-to-day Lisp programming.

This matches my experience. The barrier between application and
language/compiler development disappears, and instead you get a rich new
feedback loop between the two. Overall complexity decreases (<\-- a big deal),
and many hard things become easy.

The effects of this compound over time, so what starts as a mere notational
difference turns into a deeply different programming style, one I find so
enjoyable that doing without it feels like going back into a straitjacket
after escaping.

------
rdtsc
> The act of describing what you want the machine to do is interleaved with
> the machine actually doing what you have described, observing the results,
> and then changing the description of what you want the machine to do based
> on those observations.

That sounds really powerful and interesting, but wondering how often is that
used in practice? I can imagine maintaining and understanding a large self-
modifying program like that could get a bit complicated for a team.

~~~
orthecreedence
I think he's describing interactive programming, not necessarily self-
modifying programs. With that said, yes interactive programming is incredibly
useful and I do it all the time.

It really shines on apps with a lot of state (such as a game) where normally
you'd have to quit, change the code, recompile, run the app, and reproduce the
original state as closely as possible. With lisp you can just replace the
function you are debugging while the app is running.

That said, lisp can be used for self-modifying programs pretty easily. The
idea that the default data structures it manipulates are the same as the code
itself makes it easy to build data structures with the purpose of being evaled
(not saying this is ever a good practice).

And essentially, macros are self-modifying programs that run before the actual
program runs.

~~~
rdtsc
> I think he's describing interactive programming, not necessarily self-
> modifying programs. With that said, yes interactive programming is
> incredibly useful and I do it all the time

I do it too in python via ipython. Maybe what the author meant is "Why
exploratory programming using a REPL is great".

------
agumonkey
Similarly
[http://www.defmacro.org/ramblings/lisp.html](http://www.defmacro.org/ramblings/lisp.html)

------
pjmlp
One issue with many coders not getting Lisp, is that environments open source
environments feel short of classical Lisp enviroments like LispWorks and
Allegro CL 9.0.

Having used the old Smalltalk and Oberon environments, sadly not Lisp ones, I
really thing many still don't get it.

------
igravious
(I have a subtle optimization for S-expression syntax)(I am surprised nobody
ever thought of it)(When S-expressions are in a sequence use an extra
(special) delimiter plus the regular token separator to separate
expressions)(Maybe use dot? (period I think some call it))

Like so. I think it could catch on. And you get rid of so many round bracket
block delimiters (at least for S-expressions on the same level. for nesting
you obviously need them) that it makes reading a lot easier. Also make the
language modal and have the default be the indicative mood. Maybe replace "."
with "?" for interrogative mood? Maybe elide ".)" to ")" as another
optimization?

~~~
lisper
You'll love my binding-block macro then. You can find it here:

[https://github.com/rongarret/ergolib](https://github.com/rongarret/ergolib)

~~~
igravious
Interesting. Now all I have to do is learn a bit of Common Lisp. I'm surprised
my half-jokey question turned out to have a legitimate response :)

------
platz
> S-expression syntax is a well-designed serialization format

I must press, well-designed according to what _measure_? OP presents the well-
designed serialization format as a strict positive instead of the truth: a
tradeoff compared to other unnamed qualities.

------
jgrant27
I haven't been here in a while now but when Ron shows up in the mainstream
that's reason enough. Why Lisp ? Definitely the reasons he points out.

I worked at a _large_ e-commerce retailer for a few years, one that you might
have heard of or even purchased something from. I had the experience of
building a few systems in Lisp and also in a few mainstream imperative
languages. It was tough and really lonely even though I had real success.

With Lisp I could do things that whole teams could only dream of in Blub and
in much shorter periods of time.

The problems weren't really technical but more psychological and social when
it came to Lisp. Big problems. I wasn't very successful in resolving many of
them.

I still use Lisp here and there but have been drawn to the ML family in the
past few years. Strong static type systems got a hold of me for better or
worse.

All languages suck, it's just the degree that differs.

------
reddytowns
For me, I think the most important reason for working with lisp and its
variants is that it allows you to (and you should) code from the middle.

When I first started programming, and for a long while, I'd think top down.
What do I got? Then, how can I iterate through that to do the operations I
want and then build up what I need?

With lisp, the ability to reuse code in closures is so easy, you can afford to
think a different way. What are the most basic, clean and simple, operations I
need to do? (not necessarily in order I need to do them) This includes
conversion, summing results, iterating through a structure, etc.

After writing functions for those, you can quite easily plug them all together
without much thought.

You end up with code that is non-redundant, clear, reusable, easy to debug,
and flexible.

------
serve_yay
There are certainly good reasons why lisp is the way it is. I dislike reading
that style of code, though. I don't like how you have to read it in a strange
sort of top-to-bottom-but-also-inside-out way (really this happens in all
languages but it's especially bad in lisp because there are no infix operators
and so forth), and of course the common gripe about all the parens.

And to be honest, I am not really interested in how hard it is to write a
compiler for the language I use.

~~~
weavie
Clojure has a threading macro built in
([http://clojuredocs.org/clojure.core/-%3E](http://clojuredocs.org/clojure.core/-%3E))
which lets you write code like :

    
    
        user=> (-> "a b c d" 
                   .toUpperCase 
                   (.replace "A" "X") 
                   (.split " ") 
                   first)
    

It is trivial to write the macro in other Lisps.

~~~
sgrove
This is one of the nicest macros - chaining with sane return semantics and no
overhead/boxing required. Breaks my heart trying to replicate it.

------
icosahedron
Which lisp do you use? I've used SBCL and tried using mocl and ECL on mobile
platforms, but without any real luck. Too much work getting them set up.

~~~
orthecreedence
Clozure CL is easy to install/run on just about any desktop platform (windows
included). On *nix, SBCL is a good choice.

I've heard good things about MOCL but haven't tried it myself, and I know
getting ECL working on mobile is an uphill battle (but achievable if you have
the time). ECL switched maintainers recently, so maybe mobile is something
they will focus on in the future.

~~~
WalterGR
Seconding the Clozure CL recommendation. It supports threads on Windows, which
really expands your options if you want to do web development in Common Lisp.

I run Linux on my web servers but develop from a Windows desktop. CCL works
great on both.

~~~
lisper
Thirding CCL :-) The built-in IDE and ObjC integration are awesome for
developing on a Mac. And you can deploy finished code on Linux and Windows.

~~~
TeMPOraL
Fourthing CCL - it's stable on Windows and fast enough for writing games :).

Also, it's the best free CL for targeting Raspberry Pi - unlike SBCL, CCL
supports native threads there, and with the weak but quad-core processor of
RPi2, the difference is very noticeable.

------
hitlin37
If the page doesn't download for anyone, here is archive snapshot from today:
[http://web.archive.org/web/20150508012536/http://blog.rongar...](http://web.archive.org/web/20150508012536/http://blog.rongarret.info/2015/05/why-
lisp.html)

------
kazinator
I'm hacking on TXR these days which contains a Lisp dialect called TXR Lisp.
Lisp hacking and research is fun, in particular if you have the freedom of
your own dialect.

The current public release TXR Lisp still has an embarrassingly shoddy
implementation of "places": expressions which not only evaluate, but serve as
assignable locations. I implemented most of the place-manipulating operators
(set, inc, push, ...) as special forms, and only a small repertoire of places
is hard-coded in the interpreter. (And that, by the way, creates one of the
impediments against going to a compiler.)

I have thrown all that out and created a macro-based system for places.
Instead of copying the Common Lisp one, I drummed up something new.

Common Lisp lets programmers register assignment places as "setf expanders"
which return five values: lists of temporary variables, access and store forms
and such. (Google for "CLHS get-setf-expansion").

I have taken a somewhat different, though necessarily closely-related
approach. The creator of a new place provides two or three functions. These
functions take a place and a piece of code, and wrap it with macrolets for
accessing the place. These macrolets are given names that the caller
specifies. Of course, I have a macro for writing these functions as a capsule.

The three functions provide a way for handling the access and update of a
place, a simple store to a place without an access, and the deletion of a
place.

This third item is new. In TXR Lisp, there is a del operator: a place can be
vaporized. Not all places support deletion, but many do. For instance you can
do (del [some-vector 3]) to delete an item from a vector. Or a range. Hashes
support deletion. (del (car x)) works: what happens when you delete the car of
a cons cell is that the cdr is popped, and the popped item is transferred into
the car. Thus if x holds (1 2 3), the (2 3) part is popped to produce (3) and
the 2 moves into the car, resulting in X holding (2 3). ( _I just realized
that we need place insertion to complement place deletion!_ )

Here is how a (vecref <vector> <index>) place is defined:

    
    
       (defplace (vecref vector index :whole args) body
         (getter setter
           (with-gensyms (vec-sym ind-sym)
             ^(rlet ((,vec-sym ,vector)
                     (,ind-sym ,index))
                (macrolet ((,getter () ^(vecref ,',vec-sym ,',ind-sym))
                           (,setter (val) ^(refset ,',vec-sym ,',ind-sym ,val)))
                   ,*body))))
         (ssetter
           ^(macrolet ((,ssetter (val) ^(refset ,*',args ,val)))
              ,*body))
         (deleter
           (with-gensyms (vec-sym ind-sym)
             ^(rlet ((,vec-sym ,vector)
                     (,ind-sym ,index))
                (macrolet ((,deleter ()
                              ^(prog1 (vecref ,',vec-sym ,',ind-sym)
                                      (replace-vec ,',vec-sym nil
                                                   ,',ind-sym (succ ,',ind-sym)))))
                  ,*body)))))
    
    

[[ Digression: what is rlet in the above?

The rlet macro is something I just invented days ago. It is like let, but it
detects and optimizes cases when a symbol is bound to another symbol, and
turns these into symbol macrolets. Demo:

    
    
      $ ./txr -p "(sys:expand '(rlet ((a b) (c (form))) (list a b c)))"
      (let ((c (form))) (list b b c))
    

Look, the (a b) disappeared in the form expansion, and b was replaced by a.
That's because the macro expansion is this:

    
    
      $ ./txr -p "(macroexpand '(rlet ((a b) (c (form))) (list a b c)))"
      (symacrolet ((a b)) (let ((c (form))) (list a b c)))
    

rlet also propagates simple constants, as shown by the variable d:

    
    
      $ ./txr -p "(sys:expand '(rlet ((a b) (c (form)) (d 42)) (list a b c d)))"
      (let ((c (form))) (list b b c 42))
    

Of course, let can only be replaced by rlet in certain circumstances, when we
don't rely on the binding actually providing storage semantics. Above, we can
only replace a with b, if a is not assigned anywhere. And since d is replaced
with 42 blindly, it _cannot_ be assigned.

Obviously, rlet is something you don't need if you have an optimizing compiler
which eliminates useless temporary variables and propagates constants! I just
wanted nicer expansions from the places system.]]

So anyway, the defplace creates three functions which provide advice for how
to correctly access, update and delete a place. They wrap this advice, which
takes the form of local functions or macros, and any additional lexical
definitions they need, around a supplied piece of code, and that code can
refer to that.

Here is how this advice is used, in the implementation of the swap operator.
This shows you the real beauty of the system, because we can write a
completely robust swap, but we don't have to use anything resembling CL's get-
setf-expansion and its five cumbersome return values:

    
    
      (defmacro swap (place-0 place-1 :env env)
        (with-gensyms (tmp)
          (with-update-expander (getter-0 setter-0) place-0 env
            (with-update-expander (getter-1 setter-1) place-1 env
              ^(let ((,tmp (,getter-0)))
                 (,setter-0 (,getter-1))
                 (,setter-1 ,tmp))))))
    

What is the env doing there? We have to pass the macro-time environment down,
because places can be macros! For instance what if we do (swap a b), but a is
actually a lexical symbol macro for (car x) and b is something similar: the
with-update-expander macro will take care of expanding these places, so it
fetches the correct advice for the expanded places.

So note how we have a completely naive looking pieces of code, a three point
swap:

    
    
              ^(let ((,tmp (,getter-0)))
                 (,setter-0 (,getter-1))
                 (,setter-1 ,tmp))))))
    

All we did was generate some gensyms, use a couple of macros to obtain the
place-accessing expertise from some generated lexical helpers, and then wrote
a straightforward swap.

    
    
      $ ./txr -p "(sys:expand '(swap a b))"
      (let ((#:g0044 a)) (sys:setq a b) (sys:setq b #:g0044))
    
      $ ./txr -p "(sys:expand '(swap (car c) (cdr c)))"
      (let ((#:g0044 (car c))) (sys:rplaca c (cdr c)) (sys:rplacd c #:g0044))
    
      $ ./txr -p "(sys:expand '(swap 1 2))"
      ./txr: unhandled exception of type eval-error:
      ./txr: during expansion at string:1 of form (swap 1 2)
      ./txr: message: form 1 is not syntax denoting an assignable place
    
      $ ./txr -p "(sys:expand '(swap [f x] [g y]))"
      (let ((#:g0067 [f x])) (let ((#:g0056 [g y])) (let ((#:g0044 #:g0067))
      (sys:setq f (sys:dwim-set f x #:g0056)) (sys:setq g (sys:dwim-set g y #:g0044)) #:g0044)))
    

Note the [] notation is completely generic, so f and x could be lists (or
vectors, hashes, strings). In this case, f and x are themselves expected to be
places. Thus [f x] must be a place, and f also must be a place. The reason is
that if we manipulate a list, we may have to assign the new list to the old
variable. In general, sys:dwim-set could cons up a new list.

So in summary, Lisp is always fresh and keeps me interested in hacking. Even
old things (solved problems like assignment places) can be looked at in a new
light.

Check out this recursive implementation of shift, which is like Common Lisp's
shiftf:

    
    
      (defmacro shift (:env env . places)
        (tree-case places
          (() (eval-err "shift: need at least two arguments"))
          ((place) (eval-err "shift: need at least two arguments"))
          ((place newvalue)
           (with-update-expander (getter setter) place env
              ^(prog1 (,getter) (,setter ,newvalue))))
          ((place . others)
            (with-update-expander (getter setter) place env
              ^(prog1 (,getter) (,setter (shift ,*others)))))))
    
      $ ./txr -p "(sys:expand '(shift a b c d))"
      (prog1 a (sys:setq a (prog1 b (sys:setq b (prog1 c (sys:setq c d))))))

~~~
teddyh
> _The rlet macro_ [in the above code] _is something I just invented days
> ago._

This _perfectly_ illustrates both the power and danger of Lisp; also known as
“The Lisp Curse”¹

①
[http://www.winestockwebdesign.com/Essays/Lisp_Curse.html](http://www.winestockwebdesign.com/Essays/Lisp_Curse.html)

~~~
Guvante
> The expressive power of Lisp has drawbacks. There is no such thing as a free
> lunch.

This line doesn't hold on its own but as the ending of the article it was very
powerful.

I had heard that extra power can make it easier to do things wrong, but the
idea that extra power can cause problems even when you do things right is
fascinating.

------
myth_buster
Here's [0] PG's take on it.

[0]
[http://lib.store.yahoo.net/lib/paulgraham/jmc.ps](http://lib.store.yahoo.net/lib/paulgraham/jmc.ps)

------
albatross_down
Cached version here:
[http://webcache.googleusercontent.com/search?q=cache:KvLXVU9...](http://webcache.googleusercontent.com/search?q=cache:KvLXVU9l_0wJ:blog.rongarret.info/2015/05/why-
lisp.html+&cd=1&hl=en&ct=clnk&gl=us)

------
smizell
I actually created a hybrid of JSON and Lisp called Geneva [1]. It was a fun
weekend experiment. I totally relate to how the author feels

[1] [https://github.com/smizell/geneva](https://github.com/smizell/geneva)

~~~
PuercoPop
Early versions of wat-js used ["the", "same", "syntax"]. You might find it
interesting. [https://github.com/manuel/wat-js](https://github.com/manuel/wat-
js)

~~~
smizell
Ah, very interesting! Thanks for the link.

------
lessthunk
Why Lisp? Because the developers that know and love Lisp are normally highly
above average and get things done. For sure, there is a much smaller pool to
chose from, but once you are far from what is currently hip, you get
developers that actually care.

------
jahnu
What are some of the best known applications written in a lisp?

------
pshc
Nah, Lisp is nowhere near the most minimal way to serialize code. It's
plaintext, so it's denormalized. Something akin to bincode is the most
minimal. But that requires a non-plaintext editor...

------
dptd
Nicely said. Thanks!

------
dschiptsov
There are whole books which answers that question. The one I am aware of is
pg's "On Lisp" which explains everything clearly without any "Haskellish or
monadic mysticism".

There is also the famous essay "Beating an average" and this very site as a
walk of their talk.

There are also SICP and PAIP and Norvig's Lisp Style Guide.

Read the best thinkers.

------
dschiptsov
My 2₹ - [http://karma-engineering.com/lab/wiki/Bootstraping](http://karma-
engineering.com/lab/wiki/Bootstraping)

------
_random_
Death by 1000 nested round brackets.

