
The Machine That Builds Itself: The Strengths of the Lisp Languages (2016) - kuwze
https://arxiv.org/abs/1608.02621
======
olavk
There is a saying about the band Velvet Underground - that not many people
bought their records, but everyone who bought one went out and started their
own band. I think Lisp is kind of similar - its influence is much larger than
its fame, but ever language designer is inspired by it to some extent.

I do think Lisp kind of missed the boat when a wave of dynamic languages
(Perl, Python, Ruby) went mainstream. Today the most exciting developments in
language design seem to be around static type systems, type inference etc.

~~~
yogthos
I think there is a lot of exciting stuff happening in Racket and Clojure
around contract systems right now with things like Racket Contracts and
Clojure Spec. This approach provides an appealing alternative to static typing
because it focuses on semantic correctness while static typing focuses
primarily on self consistency.

In my view the former is much more valuable because ultimately we want to know
that the code is doing what was intended, and static typing doesn't appear to
be an effective tool for encoding intent.

~~~
stcredzero
_static typing doesn 't appear to be an effective tool for encoding intent_

Static typing is about as effective at encoding intent as intention revealing
names. That is to say, if you are diligent and clever in how you structure and
name things, a system can be very intention revealing.

HN user jerf has a method of revealing intent through types in golang, which
he calls something like microtyping. Basically, everything has its own type,
based on its domain semantics. So if you're using a float64 to store time
deltas, you don't just use the float64. Instead, you base a TimeDelta type on
float64.

    
    
        type TimeDelta float64
    

Then golang allows you to attach methods to that type, which can then provide
more opportunities for intention revealing. Also, this prevents a programmer
from mistakenly using a float64 representing width and shoving it into a time
calculation. Is this as ironclad as Eiffel Design by Contract or Haskell
typing? No. However, I've been using it myself, and I can affirm that yes, the
compiler will warn you about some semantic errors if you do it this way.

The big problem with the Intention Revealing Names pattern, is that there is
no enforcement and that it is subjective. For this reason, it's likely to fall
apart in a large enough project that lives long enough.

~~~
yogthos
The kinds of errors where you have a type mismatch are wholly uninteresting in
my experience, and get caught very early on in the development process. What I
really care about is semantic correctness.

For example, consider a sort function. The types can tell me that I passed in
a collection of a particular type and I got a collection of the same type
back. However, what I really want to know is that the collection contains the
same elements, and that they’re in order. This is difficult to express using
most type systems out there. Even when the type system is powerful enough to
encode such semantics, it's both difficult to figure out how to express and
understand the resulting specification.

~~~
smadge
I was curious about your example of expressing and validating that a sort
function returns a sorted list using the type system. I found this paper [0]
by McBride, McKinna, and Altenkirch about using dependent types in the
language Epigram to do exactly that. I have only scanned it, and it remains to
be seen if it truly is too "difficult to figure out how to express or
understand the resulting specification." Hopefully with more research it will
only become easier!

[0]
[http://www.cs.nott.ac.uk/~psztxa/publ/ydtm.pdf](http://www.cs.nott.ac.uk/~psztxa/publ/ydtm.pdf)

~~~
yogthos
The fact that it's a research topic really says all you need to know in my
opinion. It's not a question of whether you can do it in principle, but rather
whether it's an effective alternative to other approaches.

~~~
smadge
I agree, I don’t think dependent typing and validating the correctness of your
program using formal proofs is ready for mainstream adoption. I think it might
be ready for some domains, like aeronautics, where the cost of errors is very
high.

------
daniel-cussen
It's been so sad to see that while the AI winter ended, the Lisp winter that
started around the same time never did. Maybe it's coming back after
consolidation on the languages end: SBCL and Clojure are coming ahead as clear
winners, and are quite powerful.

I'm sure I'm not the only guy who started learning Lisp after reading Paul
Graham's essays, and I'm going to keep using Lisp for several reasons, but the
momentum just isn't there.

~~~
workinfunk
Is SBCL winning in anything? One of my acquaintances, a Lisp zealot, convinced
me to jump in and make my next project in SBCL. Short story, it was a joke.
Nobody seems to have done any serious work in SBCL in over a decade. Basic
things are either completely missing or broken. Clojure I can at least make
working software in. The fact that you are putting SBCL up there with Clojure
makes me wonder if you've ever used SBCL.

~~~
zodmaner
>Is SBCL winning in anything?

It is one of the most active implementation of Common Lisp (other being
Clozure CL) and one of the fastest thanks to its support of optional type
hinting (which it had long, long before gradual typing become in vogue
recently). With some type hinting, it is possible to generate a code that's
almost as fast as C (in certain cases, of course, but still much better than
other implementations).

>Nobody seems to have done any serious work in SBCL in over a decade.

Could you please elaborate on what you mean by this?

SBCL releases a new version that contains both enhancements and bugfixes every
couple of months, with the latest version (version 1.4.9, released on June 28,
2018, ~5 days ago) came one just one month after the previous version, 1.4.8,
so I really don't understand what you mean when you said that no one "seems to
have done any serious work in SBCL in over a decade".

You can also see all previous releases and their changlog here:
[http://www.sbcl.org/all-news.html](http://www.sbcl.org/all-news.html).

Or did you meant to say that no one use SBCL to do serious work? Well, with
some googling, I'm sure you'll be able to find that, while not many, there are
companies that use SBCL (and other Common Lisp implementations) in production
and other non-trivial works (not to mention various freelancers who use SBCL
to put foods on the table). Some of those companies are also involves with the
development of SBCL as well.

>Basic things are either completely missing or broken.

Care to list them? One of the strength of SBCL is that the devs are very
responsive, especially to bug reports, so I'm sure they would be more than
happy to fix the problems that you encountered.

~~~
workinfunk
Sure, here's 2 off the top of my head: Package management is a non-thing in
SBCL. It just doesn't exist. Have fun downloading 6-to-12 year old zipballs.
The basic HTTP server, hunchentoot IIRC, crashes after it serves its first
request. Hope your favicon.ico was a good one!

~~~
zodmaner
>Package management is a non-thing in SBCL. It just doesn't exist.

When is the last time you use SBCL, or Common Lisp in general? Quicklisp
([https://www.quicklisp.org/beta/](https://www.quicklisp.org/beta/)), which is
a modern package manager that supports resolving dependency and works across
almost all currently active Common Lisp implementations (and not just SBCL),
has been a thing for years now.

>The basic HTTP server, hunchentoot IIRC, crashes after it serves its first
request

Would love to see the backtrace to see the reasons (and to submit a bug report
if the issue warrant one), as from my experience Hunchentoot is very stable. I
regularly used it in my freelance jobs without any issues.

------
ectospheno
I find the Lisp variants to be lovely right up until I want to package and
ship an actual application to a normal human being. Then they are quite
clumsy. I find the same to be true of a number of other more recently popular
high level languages. I've always assumed it was because graduate students
never got to the point where they ship anything.

~~~
lgessler
I haven't tried doing this in other Lisps, but it's really easy if you're
using Clojure's most popular build tool, Leiningen. For JVM Clojure you can
produce a single .jar ("uberjar") that you can run as you'd expect, and for
ClojureScript, Google's Closure Compiler is used to output a single minified
and tree-shaken JS file.

~~~
Oreb
The parent poster was talking about shipping "an actual application to a
normal human being". A .jar file is a non-starter, as normal human beings
these days don't have Java installed and have no desire to install it.

Don't get me wrong: I love Clojure, and it's currently the language that pays
my bills. But I'd never consider using it to build an app for end users
(except possibly a ClojureScript React Native app for mobile devices).

------
zaarn
LISP always makes me sad. It's a language from which all modern languages
could learn so much (and currently a lot of features of LISP get cloned). LISP
Machines pioneered basically 90% of what modern computers and the internet do
today.

Yet, it's a very niche language, an almost forgotten artifact of time.

~~~
eafkuor
Clojure for me is the perfect modern LISP, too bad it seems it's dying (happy
to stand corrected there)

~~~
edem
You seem right. Every metric I can check says it is going downhill. Too bad, I
used to go to the Clojure meetups in my area, I even had a talk about Clojure
+ Emacs.

~~~
xixixao
Many reasons. S-expressions would be the least of my worries. Java is dying,
not on the whole, but certainly in mindset of people who are picking new
languages. It’s integration with OSs, which Clojure depends on, is arcane by
today standards. Second, it’s tightly controlled by one person, and they
happen to hate types. Similar to CoffeeScript, couple years after the language
inception people figure out that they miss types. Adding them back in is super
tough.

And third, abstractions have costs. There is no wide-spread popular LISP, and
people should really ask themselves why. We have lots of code generation, and
depend on it more and more. But it’s “simple” (or “primitive”). It’s usually
one step. You can read the input, and you can read the output.

I’m starting to get a sense that macros are like currying. Elegant, beautiful,
powerful. But net negative if your concern is to get lots of people to create
software together.

~~~
realusername
> Java is dying, not on the whole, but certainly in mindset of people who are
> picking new languages

Even as a whole it's slowing down heavily now, even Android is switching to
Kotlin. The language is already not that trendy but Oracle's aggressive
lawsuits is making it even worse.

~~~
knightofmars
Nope.

[https://www.tiobe.com/tiobe-index/](https://www.tiobe.com/tiobe-index/)

~~~
realusername
I did not say that it's still not used heavily, just that it's slowing down.
And C is the second on that list so I'm not sure I would believe this ranking,
I have not seen much C jobs for a while now.

~~~
knightofmars
"I have not seen much C jobs for a while now."

Because your anecdotal evidence is more accurate than what this list (which
describes its methodology) is showing you?

~~~
realusername
Yeah it is, because their methodology is shitty: [https://www.tiobe.com/tiobe-
index/programming-languages-defi...](https://www.tiobe.com/tiobe-
index/programming-languages-definition/), they just use the google popularity
of "[name] programing", it's not a mesure of anything.

C jobs are very rare, and it's certainly not 14% of the programming jobs.
Putting Java and C on the exact same level is a good sign something is wrong
with what they are doing.

~~~
kickopotomus
C jobs are not rare at all. I think you are failing to consider large chunks
of the tech sector that operate outside of the web.

~~~
realusername
> that operate outside of the web.

Outside of the web, mobile, desktop apps and even gaming (mostly C++) you
mean? C is used mainly in low level computing and embed nowadays.

~~~
kickopotomus
Yes. That is correct. I fail to see your point since that is a large chunk of
tech.

~~~
realusername
I would not use "large" for that, it's certainly a chuck yes but is it
comparable to web, mobile, gaming and desktop? I'm not really sure.

~~~
kickopotomus
It most definitely is large. It's not just about all of the recent IoT
devices. Firmware for all of the controllers on your motherboard is written in
C. Most device drivers are still written in C. Many operating systems (linux)
are written in C. Then you have all of the controllers in the devices in your
home (e.g. microwave, oven, digital clock, entertainment devices, etc.)

------
phlakaton
I'm disappointed by this paper. The general review of the language is of the
sort that you can find a dime a dozen on LISP enthusiast sites--so this paper
does little to effectively recruit existing programmers to the cause. On the
other hand, the references to current bioinformatics projects using LISP is
helpful as an index, but provide few details to convince a biologist that
they're worth checking out--so I'm not convinced on that side either. The "key
point" of a forthcoming BioLisp is embarrassingly glossed over.

I think the paper would be much stronger if it focused on that BioLisp aspect,
talking about:

* What is BioLisp? A mere web portal pointing to various resources across various languages? An attempt to create a curated installation of version-compatible libraries for a specific LFL? An attempt to develop a fully-integrated framework for that LFL??

(I think Sage for Python might be a useful model here, but if you were trying
to build that, you'd have to start by picking your LFL, which the authors at
present seem unwilling to advocate. That in itself demonstrates one of the
main liabilities at present in adopting LISP--the community is fractured
between Racket, Scheme, Clojure, and various implementations of Common LISP--
often to the extent that they argue over whether their respective opponent
languages even _deserve_ to be called LISPs. And, of course, in this forum we
even have Arc fanatics to add to the lot. ;-) )

* What challenges face biologists attempting to use LISP in their work? How does BioLisp help to address those? ("Your language lacks macros" is not an argument at a sufficiently detailed level, I'm afraid. Rather, actually show them a DSL built in LISP that they will desperately want!)

* For that matter, what challenges face biologists attempting to use R/Python/Julia/etc for their work? How does BioLisp help to address those?

* What promising pieces could BioLisp be built upon? What pieces are missing? What pieces need modernization?

By all means, I encourage scientists and LISP enthusiasts to make the case
that biologists should/could be using LISP--but make sure you actually make
the case!

------
mark_l_watson
Great writeup on specific advantages of Lisp languages! I have used Lisp on
two long term medical and bio information consulting projects (one also used
semantic web/linked data tech) and I could not agree more that building Lisp
up to be a new language for applications, repl based development, and the
expressiveness of Lisp languages are all huge wins in building knowledge
intensive systems.

EDIT: I will add that although I enjoy using Lisp languages, and have written
a few Lisp books, that I don’t like to talk anyone into using any particular
programming language or technology stack. There is a difference in enjoying
our own choices to telling other people what choices they should make.

~~~
marmaduke
What do you do for heavy number crunching and the like?

~~~
mark_l_watson
In ancient history I used FORTRAN. For machine learning, which is my job, I
use TensorFlow using the Keras APIs. There are a few interesting machine
learning libraries in Common Lisp but I don’t use them professionally. I am
experimenting with saving trained Keras models and converting them to a format
with runtime code for Racket Scheme - I might open source that if I can
improve the performance.

~~~
TheRealKing
In modern history, I still use (modern) Fortran (as well as MATLAB, Python,
and R). It does what I need to do, efficiently with minimal coding.

------
arknave
> "Lists, which are a generalization of graphs, are extraordinarily well
> supported by Lisp."

I always thought of linked lists as a special kind of graph, where each node
except the head and tail had exactly one incoming edge and one outgoing edge.
What's a better way to think of this in terms of LISP?

~~~
thaumasiotes
Internally a Lisp list is a binary tree in which left children are leaves
storing a value and right children aren't leaves (except the terminus) and
don't store values.

Or rather, there is a structure called a cons cell which consists of two
pointers, "left" and "right"; a cons cell in which the right pointer is NIL is
a list; and a cons cell in which the right pointer points to a list is also a
list. The values in the list are whatever is pointed to by the left pointers.

If you diagram this out, you'll find that your mental image is basically
accurate already. It seems difficult to describe lists of this form as a
"generalization" of graphs.

------
carapace
All these comments about how Lisp missed the boat, and you're all wrong. _We_
missed the boat. Lisp has been going strong among it's own the whole time.
It's not their fault we're all so stubborn, punishing ourselves over and over
again like some sick masochists.

I got _mad_ when I finally learned Lisp and realized how much time and effort
has been wasted over the years because we're too lazy, ignorant, or foolish to
use it.

(As a tangent, it's simply incredible how deliberately ignorant programmers
can be. I've worked with guys getting paid to write software who didn't know
who Alan Kay is, or had never heard of Prolog. What's worse is they're not
ashamed of their ignorance. Can you imagine a physicist who had never heard of
Newton?)

I don't use Lisp, but I have the grace to admit that that's a _personal
failing_. (I joke that Python's only major problem is that it ruins you for
other languages... I just can't quit that sweet syntax. Although Python 3 is a
train wreck IMHO. Don't get me started.)

I _would_ switch to Lisp, but I found something even better, which brings me
to my actual, non-ranty, point:

 _There 's a language even better than Lisp._

It's called Joy and it combines all the best parts of Lisp with the best parts
of Forth to make an enormously simple system for describing software. It's a
purely functional, stack-based, "concatinative" language that turns out to be
good for "Categorical" programming: programs in Joy can be "instantiated" over
different categories to generate different kinds of computations. I only have
one good reference for "Categorical" programming right now: a paper and talk
by Conal Elliott "Compiling to Categories" February 2017
[http://conal.net/papers/compiling-to-
categories/](http://conal.net/papers/compiling-to-categories/) He's working in
Haskell, not Joy, but he's describing the idea: from one piece of code you can
get calculations, dataflow diagrams, circuit descriptions, type signatures
(inference), derivatives, etc., by implementations of the categories for each
kind.

------
Areading314
As elegant as lisp is, I think the flexibility just kills its usability on
teams, while more rigid languages like Java, Go or C see massive adoption.
Language expressiveness is rarely a real bottleneck, compared to debugging and
understanding program flow and getting new contributors up to speed.

Separately I think each lisp has an individual fundamental flaw that makes it
unusable for production, eg clojure having terrible error handling, common
lisp being case insensitive, scheme having a minimal standard library, racket
being fairly slow, etc.

~~~
junke
I tend to find it very easy to debug CL code. In fact, CL is designed to be
introspective: "inspect", "trace", "break", "disassemble" and "step" are part
of the language.

Also, Common Lisp is case sensitive.

~~~
workinfunk
You can hardly fault newcomers for not understanding the finer nuances when
everything they type automatically gets uppercased.

~~~
junke
That's why I try to correct people when possible and point newcomers to
Practical Common Lisp (Seibel).

------
0vermorrow
The paper itself doesn't focus on Machine Learning / AI or anything too
specific for that matter (it reads like a review / praise of LISPs), but I am
currently researching Symbolic Artificial Intelligence as well as let's say
the history of "Good Old-Fashioned Artificial Intelligence" and the role LISP
took in all of that.

Any books / papers / articles that one should definitely read on these topics?

~~~
zshrdlu
You may be interested in a little-known older book _Computational Semantics:
An introduction to Artificial Intelligence and Natural Language Comprehension_
:
[https://archive.org/details/computationalsem04char#mainconte...](https://archive.org/details/computationalsem04char#maincontent)

------
oldandtired
What are the pros and cons of Lisp (or any other language for that matter)?

In my undergraduate days (oh so many decades ago), I was introduced to Lisp.
For some of the problems we were solving, it was the perfect tool. For others,
a different language could be used.

The pros of Lisp (in all its varieties) are in many ways the cons of Lisp.

The macro system is both beneficial and antagonistic to the development of
sane programs. Used judiciously, macros may life writing code much easier.
However, it is not uncommon for Lisp programmers to write macros without
consideration for those who come after them. This is, of course, true for any
language.

There are some major systems out there that are written in Lisp and when
trying to change them (in a maintenance way), it is just as painful as in
pretty much any other language.

The competent and talented Lisp programmers is imbued with the same arrogance
that C++ competent and talented programmers have. Tricky programming is fine
if it is fully documented and the reasoning for the tricks is fully explained.

Lisp and its variations have much to offer the programming community. But the
way programs are written quickly diminishes the effectiveness of the language
for the larger community.

Each language has features that can be of use to each of us. they are tools we
can use to solve the problems we are facing. No one language is suitable for
all problems. If you have that attitude then you only see hammers and nails.

------
AzzieElbab
We have reached the point where saying "pick the right tool..." usually means
completely opposite. PHP for web servers, python for large applications, go or
anything c like for business logic

------
wodenokoto
What's a good lisp ide other than emacs?

To me, learning lisp and Emacs at the same time seems like a tall order

~~~
kkleindev
Why are you only looking into ides? Any editor might do a solid job. When
working with clojure, parinfer[0] was all I really needed.

[0]
[https://shaunlebron.github.io/parinfer/](https://shaunlebron.github.io/parinfer/)

~~~
wodenokoto
Since everything is a function, I really, really want inline function argument
hints.

------
transfire
You underestimate the power of the Forth.

------
magicmouse
Lisp is part of the family of languages that J. Backus referred to as
'transformational languages'. Its ability to self-modify creates programs that
are almost impossible for people other than the author to understand. This is
why after 50 years it is still a niche language, and will remain so. There is
little demand for programming languages that are hard to maintain. A working
program lasts decades, and passes through many hands, and LISP is dead-end
technology that will forever keep being brought up as its temptations are
irresistible to some.

~~~
sametmax
I don't get the downvote. This is my experience too. You browse a code source,
unable to understand some of it. You google it and find nothing. Only to
realize that the author defined it's own little world. As usual, zero unitest
or tutorials. It makes lisp a terrible experience for beginers.

~~~
kqr
I don't share this experience. I mean, I'm absolutely sure it can happen – in
particular at shops where there's a high level of employee churn and they
throw consultants at a problem and then kick them out again. Of course you'll
get all sorts of weirdness at then.

But with some continuity and a carefully crafted own little world, everybody
on the team soon chooses voluntarily to speak in the terms of that world,
because it is a world well suited to the problem at hand.

~~~
sametmax
Those are ideal rare conditions you are talking about.

Also they don't apply to online open source projects and tutorials.

------
agounaris
I see a title "The Machine That Builds Itself"....but there is no code! Show
me the code or throw the paper away...

------
hevi_jos
The guys that wrote this piece have drunk the kool aid of Lisp.

To say that library support for bioinformatics is on the early stages is quite
a misunderstanding. Lisp was one if not the first language to be used in
everything, including bioinformatics. If it is not used anymore is because
people have abandoned it.

People have abandoned it once its flaws became evident. The main flaw of Lisp
is using linear data for everything. It is obsolete, it was designed for
linear memory like tapes or punched cards, not random access to memory like we
have today.

Languages like Clojure has not this limitation but they have others, like
forcing you to use the java virtual machine, good if optional, terrible if
mandatory.

In current systems with huge amount of RAM, when you could have 128GBs on a
developer system having to use a system that shoehorns everything to Lists is
going backwards.

It also maximizes verbosity as everything uses the same thing, pattern
differentiation is hard for actual humans to read and understand. This was a
serious flaw that even Lisp creators quickly identified and tried to solve,
remember Lisp was considered a temporary solution while the good solution
was(never) found.

Not to talk about the absence of privileged instructions, everybody on the
team changing whatever it wants, even basic instructions.

Don't get me wrong, I love Lisp, it is one of the more interesting languages
in the world. I am actually working on porting some of the features it
has(introspection) to other languages.

But from my point of view it is not that good for production for most
projects. I have drunk the kool Aid myself before and have the scar tissue
from that.

~~~
sametmax
Another problem is that lisp let you do anything. And so people do.

So everytime you get on board on a project you hit your toes on the DSL du
jour, of course badly documented and tested. It's like learning a new language
everytime, except a beta version one with no resources.

~~~
armitron
And yet another problem is people who talk about things they have no personal
experience with, resulting in them parroting something they saw or heard and
propagating blatant falsehoods. Which is what you and the person you replied
to, are doing.

Wouldn't it be better to actually inform yourselves first, by learning Lisp,
rather than regurgitating somebody else's opinions?

