
The Tragedy of the Common Lisp: Why Large Languages Explode - mpweiher
https://medium.com/@erights/the-tragedy-of-the-common-lisp-why-large-languages-explode-4e83096239b9
======
btilly
After programming for 20 years, I've noticed that programmers are aware of the
potential benefits of any abstraction that they have internalized, and
blithely unaware of the costs. Because, having climbed that learning curve, it
is now free to them.

The result is that programmers introduce complexity lightly, and when they
walk into a new language/organization/etc are inclined to add random
abstractions that they are used to from past experience.

And yes, I am not immune.

~~~
tempguy9999
Your criticism of abstractions is very much in the abstract.

What abstractions?

What costs of these? In human comprehension, in runtime, in reliability, in
mem/cpu?

Can you give examples which I can usefully learn so as to avoid?

TIA

~~~
btilly
Any and all abstractions. The cost is usually in all of the above.

Examples that I commonly encounter include OO, closures, dependency injection
frameworks, complex configuration systems, various code generation systems,
and on and on and on.

In general the tradeoff is this. For those who have internalized the
abstraction, they can think about more complex things. Those who have not
internalized the abstraction find it hard to figure out how the system works
at all until they internalize it. So when you work on code that has a lot of
abstractions under the hood it becomes either a black box (that occasionally
you dig into) or (very often) a requirement that you understand X before you
can even start to work on the code.

A rule of thumb that I use is how long the stack backtrace is. If every bug
creates a stack backtrace that is dozens of frames, there are a lot of
abstraction layers in place. And when all of the layers are actively being
worked on, it adds up - quickly.

Deciding whether given abstractions are worthwhile for a given problem
involves a judgment call. Unfortunately the people who are in the best
position to make those calls tend to be the most senior, and tend to be the
least aware of the costs of the abstractions that they add.

~~~
tempguy9999
A nice reply, upvoted.

I would note that if you start your list with OO then you're including
abstractions that almost all professional programmers would consider not
abstractions but basic tools. It's impossible to work without such
abstractions, except by working in purely procedural code, and even then...

I would add that if "a requirement that you understand X before you can even
start to work on the code" is the case then your abstraction has - arguably, I
may be wrong! - failed. My DOM iterator abstraction would take some effort of
understanding to _maintain_ , which was my concern, but it was a very simple
black box to use.

~~~
btilly
A simple example of an abstraction for which this is not true is the MVC
design. If you don't understand how it works, you don't know where to begin
looking to make a change to the system.

And yes, some abstractions truly are basic tools. And the more experience you
have, the more basic they seem, and the more such tools you have.

I am not arguing against abstraction per se. What I am arguing is that
abstractions bring a cost, and that cost adds up. A project needs to find an
appropriate balance, and there is a tendency for experienced programmers to
draw that balance at a point that may or may not be appropriate for the people
who have to maintain the system after them.

------
lacker
_The Algol, Smalltalk, Pascal, and early Scheme languages were prized for
being small and beautiful._

Those languages are also not very popular at all today. Perhaps "small and
beautiful" are not the right metrics to optimize programming languages for.

In many ways, keeping the core JavaScript language small has led to the
JavaScript ecosystem being too large and sprawling. For example, look at
module importing, something which is a fairly normal part of most programming
languages. It still works in a completely different way in Node.js and in
browsers. I think if there had been a standard way to handle JavaScript
imports years ago, the practical experience of working in JavaScript would be
simpler today.

~~~
cat199
> Those languages are also not very popular at all today.

But yet, C (arguably 'current algol') and Python (arguably 'current scheme' _)
are..

_ yes: python is verry loosely like scheme, this is meant in the sense of a
'dynamic loosely typed language you can interact with in a repl'

~~~
astrobe_
Lua is a far better counter-example as a successful small and (conceptually)
beautiful language.

~~~
cat199
yes, probably so. was going for 'popularity' here

------
threatofrain
> Once a language gets beyond a certain complexity — say LaTeX, Common Lisp,
> C++, PL/1, modern Java — the experience of programming in it is more like
> carving out a subset of features for one’s personal use out of what seems
> like an infinite sea of features, most of which we become resigned to never
> learning.

This is how I felt about Racket, and I felt I had to do an opinionated pruning
before I could use it for teaching, but the problem is that the docs don’t
have such a boundary for learners so you feel tempted to rewrite the docs.

~~~
gmfawcett
Last I checked, Racket literally has sublanguages defined for teaching and
learning: "Beginning Student", "Beginning Student with List Abbreviations",
etc.

[https://docs.racket-lang.org/htdp-langs/index.html](https://docs.racket-
lang.org/htdp-langs/index.html)

~~~
threatofrain
I wouldn't call those sub-languages, I think they're rightly different
languages (of eerie similarity) born from specific pedagogical or academic
vision. So the upkeep in dealing with the mismatch between proper Racket and
these student languages wasn't something I was interested in. Students should
be able to consult general Racket resources.

~~~
neilv
Exactly, I think that the Racket teaching languages were started to support
teaching young/new students (and not necessarily people who are or will be CS
majors) a particular way of approaching program design problems.

Racket itself arose from building a cross-platform toolset to support building
those programming tools. It also became a toolset and testbed for PL research.

I appreciate what you're saying about Racket having a big library. I was
recently talking about how to to teach Racket to _experienced_ programmers.
I'd put them down in front of a full `#lang racket/base`, but start with
teaching them only a specific subset of R7RS, and incrementally introduce a
few more concepts and exercises to try for a day or more (things you can
practice while writing your own real code, even if it briefly seems harder
than something we told you that you could use before). If instead I just
dumped the full Guide and Reference on experienced people, they'd become
productive pretty quickly, albeit colored heavily by what they previously
knew, and they might have a much longer path to becoming strong in some
unfamiliar fundamentals.

BTW, another educational thing that Rackets `#lang` does is support students
working through SICP use DrRacket. It emulates the particular older Scheme
variant used in SICP, gives an IDE maybe a bit easier for new students, and
runs on the student's own modern computer.

------
kerkeslager
I feel like this is the real success of Python: the "one and preferably only
one obvious way to do it" idea means that new ways of doing something are
rarely added, and when new they _are_ added, old ones are likely deprecated.
Complain all you want about the 2 to 3 transition, but it's resulted in a
simpler, easier-to-use language. If the community really feels a different way
is better, they add it in libraries. My only criticism of Python in this
respect is that they didn't take it far enough: Do we need higher-order
functions AND loops? Do we need classes AND first-class closures?

Common Lisp isn't even close to the most complicated language out there. Every

~~~
a1369209993
> Complain all you want about the 2 to 3 transition

Okay.

> it's resulted in a simpler, easier-to-use language.

Does `print(len("ẅ"))`[0] still produce a value (2) that is neither the
number of characters (1) nor the number of bytes (3)?

0: "print\x28len\x28\x22\x77\xCC\x88\x22\x29\x29"

~~~
kerkeslager
It took me a bit to understand what you were trying to do. Here's a paste of
my Python 3 shell session, showing that Python 3 does indeed return the number
of characters.

    
    
        ~$ python3
        Python 3.7.3 (default, Mar 27 2019, 09:23:15) 
        [Clang 10.0.1 (clang-1001.0.46.3)] on darwin
        Type "help", "copyright", "credits" or "license" for more information.
        >>> len("ẅ")
        1
    

Python 3 uses UTF 32 internally, so the byte representation you placed below
your post is not how Python 3 represents it. Instead, it looks like this:

    
    
        >>> "ẅ".encode('utf32')
        b'\xff\xfe\x00\x00\x85\x1e\x00\x00'
    

This has disadvantages (memory usage) but for most cases where Python is used,
it's an advantage (faster random access, more intuitive for situations like
the one you've proposed).

~~~
zephyrfalcon
So, I see:

    
    
        Python 3.7.3 (default, Mar 27 2019, 09:23:32)
        [Clang 9.0.0 (clang-900.0.39.2)] on darwin
        Type "help", "copyright", "credits" or "license" for more information.
        >>> len("ẅ")
        2
    

I wonder why this is? Is the Clang version relevant here?

EDIT: Your "ẅ" doesn't seem to be the same as the OP's "ẅ", although they
look the same at first glance.

    
    
        >>> "ẅ".encode('utf-8')
        b'w\xcc\x88'
        >>> "ẅ".encode('utf-8')
        b'\xe1\xba\x85'
    

EDIT 2. More info:

    
    
        >>> import unicodedata
        >>> w1 = "ẅ"
        >>> w2 = "ẅ"
        >>> unicodedata.name(w1)
        'LATIN SMALL LETTER W WITH DIAERESIS'
        >>> unicodedata.name(w2)
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        TypeError: name() argument 1 must be a unicode character, not str
        >>> unicodedata.name(w2[0])
        'LATIN SMALL LETTER W'
        >>> unicodedata.name(w2[1])
        'COMBINING DIAERESIS'
    

So the second version (w2) _does_ seem to consist of two separate
"characters", LATIN SMALL LETTER W and COMBINING DIAERESIS, which is
apparently not the same as the single-character LATIN SMALL LETTER W WITH
DIAERESIS. I guess these are actually Unicode code points and not so much
"characters" to a human reader, but as another poster pointed out, what the
number of characters should be in a string isn't always clear-cut.

~~~
a1369209993
Correct, w2 is one character (latin small w with umlaut) represented by two
unicode code points. I didn't realize there was a NFC code point for that
character; try "\x66\xCC\x88" (f̈) or "\x77\xCC\xBB" (w̻) instead.

> the number of characters should be in a string isn't always clear-cut.

This is why I use examples from latin-with-diacritics, where there is no
ambiguity in character segmentation.

~~~
kerkeslager
Interesting, I learned a bit about Unicode here. It looks like copy/pasting
combined the two code points into one when I ran my code.

Still, to the original point, I think this is more of a criticism of Unicode
than of Python. It seems to me that the answer is to not use combining
diacritics, and that Unicode shouldn't include those.

~~~
a1369209993
> this is more of a criticism of Unicode than of Python

True, although it's more specifically a criticism of Python _for using
Unicode_ , where these kinds of warts are pervasive. See also "\xC7\xB1"
(U+01F1 "Ǳ") which is two bytes, one code point, and _two_ characters with no
correspondence to those bytes.

> the answer is to not use combining diacritics

This doesn't actually work, sadly, because you can't represent eg "f̈"[0]
without some means of composing arbitrary base characters with arbitrary
diacritics.

0: If unicode has a added a specific NFC code point for that particular
character, then that's bad example but the general point still stands.

------
lerax
[hyperbolic] Common Lisp deserves more love. People are just too much jealous
to admit that CL is one of the best language of the world. How someone can
think that JavaScript, PHP, Java and other stuff alike are at least non-
miserable in comparison with something so powerful as Common Lisp?
[/hyperbolic]

Keeping the humor-ish stuff aside, actually taking CL here as example it's
someway ignorant. That's doesn't make any sense to me. Seems that all that
opinion are based in nothing more that nothing when it concerns about Common
Lisp literature experience and personal experience with the language.

Some languages are popular, but they being popular doesn't proofs that they
are good languages. This is a fact based on the type of problems most of the
people are trying to solve.

CL being big it's some way comparable to C++, but for big problems, sometimes
we need complex tools. CL is not for kids, neither C++. Common Lisp it's a
language for hackers.

~~~
nydel
my neck aches from nods in agreement. i know i’ve written at leeaasst one
#’TRAMADOL-SIMULATOR but i’ll likely just lambda the notion, slightly faster
&or funner than grepping my filesystem for “\\.l*sp$”.

------
nonbirithm
I kind of wish there was a lisp with the "recompile on error and continue"
feature of Common Lisp but without a massive standard library. A standalone
SBCL program seems to be around 40MB at minimum. It feels like it would be
doable if only CL wasn't designed with the kitchen sink included. Recently
I've gotten into Janet[0] and really like the language, although I do miss
CL's recompilation magic at times. It feels like a Lisp dialect with similar
style to Lua: tables, coroutines, small language core, embeddable as a single
C file, etc.

[0] [https://www.janet-lang.org](https://www.janet-lang.org)

~~~
lispm
SBCL's programs are 40MB because they contain all dev tools and the code is
native code, which is also large.

There are a bunch of Common Lisp implementations which are smaller (CLISP),
can create small applications (Lispworks), can be embedded (ECL) or can
compile to smallish C code (mocl).

~~~
sansnomme
Also, Common Lisp's standard library cannot compare to Python, Java etc. They
have a lot of _language manipulation_ tools but not actually anything that
will help you ship quickly if you are building generic CRUD style
applications. CL tends to shine when it comes to extremely domain specific
niche stuff where you practically have to write your own tooling from scratch.

~~~
soiheard
Roswell comes to mind for application building and distro management. Add in
ningle for web app building and woo for the server (the benchmarks are really
good), and you're basically good to go.

I think the main problem is there's too many implementations of the same thing
in CL, so devs basically have to agree on a very specific subset of tools for
every project, which adds a fair bit of overhead.

[https://roswell.github.io/](https://roswell.github.io/)
[https://github.com/fukamachi/woo](https://github.com/fukamachi/woo)
[https://github.com/fukamachi/ningle](https://github.com/fukamachi/ningle)

p.s. Not the author, but a big fan.

~~~
sansnomme
Last I tried it's not reliable. E.g. generating binaries through VM images is
coin toss.

------
callinyouin
I get that the article used Common Lisp to create the punny title and much of
its contents are broadly about why its important to keep languages relatively
small, but I'm sort of disappointed most of the language-specific talk was
about JS. I'm in the middle of learning CL at the moment (I have a mostly
Java, Python, C++ background, in that order of experience) and would be
interested to hear from experienced Lispers on why some other Lisp would be a
better choice at a language level (so not based on convenience, tooling,
popularity, etc... okay, maybe tooling).

~~~
Mikeb85
Racket seems to be favoured individuals who are using it to create a DSL or
accomplish very specific tasks, but CL is definitely the most used Lisp (well,
maybe apart from Clojure) to create 'big' applications.

------
dreamcompiler
This comparison is inappropriate. When Common Lisp was standardized, the goal
was to unify several different Lisp dialects, not to make the language
"small." Furthermore, there was no standard library for Common Lisp. The
language itself is its standard library. If anything, the language by itself
is too small for a lot of modern applications. (The situation is different
now; there are hundreds of libraries to fill in the missing pieces of Common
Lisp. But they're not standardized like the core language is.)

~~~
lispm
> the goal was to unify several different Lisp dialects

the goal was to unify several similar Lisp dialects as a Maclisp successor

------
didibus
As an aside, I recently tried Common Lisp, and it's actually pretty good,
you'd be surprised.

------
mateuszf
Isn't c++ a large language? And yet it's still used.

~~~
cjensen
Sure. But there are a lot of areas where you just shrug and say "here be
dragons" and move on.

For example, consider the new for loop syntax

    
    
      for (auto item: a().b().c()) ...
    

Does this make you nervous? It should. In the above, suppose a(), b(), and c()
are all returning objects. Which of these objects remains valid until the end
of the for loop? And this is a dragon in one of the most helpful parts of new
C++.

~~~
mateuszf
Yeah, I'm not a big fan of the language, I'm only saying it's used a lot.
Seems that in this case there was not a lot of portable fast alternatives for
system / device / high performance programming.

------
dang
Related from 2015:
[https://news.ycombinator.com/item?id=9738866](https://news.ycombinator.com/item?id=9738866)

------
didibus
I think the word Large is pretty ambiguous. In my opinion, it is more about
incoherent, inconsistent, complected, and with no clear way to learn more
about things you do not understand. A language which has everything I would
need already available and ready to use, in a simple and straightforward way
is pretty great otherwise.

------
qwerty456127
The same happens to CPU instruction sets. Old stuff should get deprecated once
better features are added (letting you write more concise and efficient code)
but you can't deprecate that as it would break too much legacy code people
rely on.

------
carapace
Awesome to see Mark S. Miller on the front page. One of my favorite computer
nerd jokes: Ted Nelson points out that Mark Miller's name is his vocation. :D
Get it? He's a symbol grinder, a mark miller. I love it.

------
amelius
What exactly is the link with the tragedy of the commons?

[https://en.wikipedia.org/wiki/Tragedy_of_the_commons](https://en.wikipedia.org/wiki/Tragedy_of_the_commons)

~~~
btilly
The word "Common" is a reference to the language Common Lisp:
[https://en.wikipedia.org/wiki/Common_Lisp](https://en.wikipedia.org/wiki/Common_Lisp).

No explicit link with the tragedy of the commons was implied. Implicitly, of
course, it applies - there are many participants, each of which is aware of
their own gain and unaware of how their actions cost everyone else. The result
is that when they all get their changes in, the result is terrible for
everyone.

For another example, C++ is a terrible language, but it contains many good
ones.

~~~
amelius
> No explicit link with the tragedy of the commons was implied.

Okay, but the article says:

> Adapted from a 2015 es-discuss thread. “Common Lisp” is not the topic. It
> serves only as one of many illustrative counter-examples.

So if "Common Lisp" is not the topic, and the tragedy of the commons is not
the topic, then it seems the title is not very well chosen.

------
gnaritas
Lisp isn't a large language, it's a small language with a large library.

~~~
nabla9
I think comp.lang.lisp had this discussion 10 years ago and the "core
language" semantics of Common Lisp is something like 25-30
functions/operators. Rest of the language could be a separated into libraries.

But even with the all those 'libraries' COMMON-LISP package has just 978
external symbols.

~~~
kazinator
In TXR Lisp, I seem to have nearly double that in the analogous public library
package called _usr_ :

    
    
      1> (len (keep-if [orf boundp mboundp fboundp] (package-symbols 'usr)))
      1713
    

That's just a one-man project coming up to ten years two months hence.

That doesn't count any structure types or their slot names, FFI types, and
local macros involved in syntaxes like _awk_ and such.

I would say that Common Lisp shows amazing restraint, given its scope and
number of people involved.

------
momokoko
_> The Algol, Smalltalk, Pascal, and early Scheme languages were prized for
being small and beautiful. _

But they did not achieve widespread adoption.

Widely used languages need to apply to a wide range of use cases. For example
the fat arrow(=>) class syntax that binds this.

    
    
        class Foo {
          bar = (baz) => {
            // do something
          }
        }
    

For entry level JavaScript developers working in React, this made one of the
most common hangups go away for them due to the automatic binding of `this`.
But for a some other situations, that might end up an anti-pattern.

That's always the issue with small, clean implementations. They work when they
only apply to one domain. Once they need to be used for different things, they
then need to add new features. It is no different for software, blenders,
bicycles etc. Once something is used for multiple domains, it now needs to
become larger and more complex. Its basically a rule of nature.

~~~
dbcurtis
> > The Algol, Smalltalk, Pascal, and early Scheme languages were prized for
> being small and beautiful.

> But they did not achieve widespread adoption.

> Widely used languages need to apply to a wide range of use cases.

A long time ago, an old-timer, even more old-time than me (and I did CS home
works on punch cards in PL/I...) was a C fanatic, and was dragging his
mainframe colleagues into Unix, said it best: "C does not get in your way."

He (and I) came from an era when eventually pretty much every language
eventually "got in your way" and you had to learn the assembly language
calling and linking conventions and implement a few nuts & bolts to get your
job done. C was the first popular language that didn't eventually knee-cap you
like that. That is why C won and Pascal was an intense but passing fad.

To me, that is the essential message for language designers, more important
than "keeping it small". Stay out of the programmer's way. Of course, there
are different ways of staying out of the way.... Python's "duck typing" is a
way of keeping the type checker out of your way. Rust's "unsafe" is a way of
keeping the borrow checker out of your way when you really, really, need to
carve with your sharpest knife.

So I would argue that "applies to a wide variety of use cases" simplifies to:
does not get in your way no matter what your application is.

As an aside: The old timer was a role model to me in one important aspect:
Learning to recognize progress when you see it. He not only lived through the
transition from core to solid state memory, he lived through the transition
from transistor logic to integrated circuits. But he was evangelizing C and
Unix when many his age thought the list of interesting programming languages
had len()==3: FORTRAN, COBOL, assembly. I took that lesson to heart, and try
very hard to answer the question: "Is this progress, or is this just
different?".

~~~
nostrademons
Go does this really, really well. I'm not a huge fan of Go-the-language
(largely because I enjoy having a little more abstractive power), but I have
to admit that one of its huge strengths is that it's a language designed to
take the focus off the language and onto the program you're writing. It
avoided several of the pitfalls of more "advanced" languages (eg. Common Lisp,
Haskell, even Rust and Python 3.5+) in the process, which is perhaps why it's
seen more mainstream adoption.

Kotlin's another language that looked at say Scala (which is an immensely
powerful language, but easily lets you get lost in abstractions), decided "We
don't need that feature unless it actively helps remove friction when writing
programs", and ended up with a lot more real-world programs written in it.

