I keep hearing sentiments like this but then I wonder, if Clojure or <another awesome Lisp> is so much better than Python <or some other mainstream language>, then why are we still writing in those languages? If it's because of libraries, then the question is: Why do people write libraries for these languages and not the Lisp ones?
Lisps are "wizard" languages: the runtime semantic is kept close to the syntax, which also means that you can rapidly extend the syntax to solve a problem. This quality is true of Forth as well, and shares some energy with APL and its "one symbol for one behavior" flavor. With respect to their metaprogramming, the syntactical approach hands you a great foot-gun in that you can design syntax that is very confusing and specific to your project, which nobody else will be able to ramp up on.
But Algols, including Python, are "bureaucrat" languages: rather than condensing the syntax to be the exact specification of the program, they define a right way to form the expression, and then the little man inside the compiler rubber stamps it when you press the run button. In other words, they favor defining a semantics and then adding a syntax around that, which means that they need more syntax, it's harder to explain the behavior of a syntactical construction, and it's harder to extend to have new syntax. But by being consistent in a certain form-filling way, they enable a team to collaborate and hand off relatively more code.
IMHO, a perfectly reasonable approach I'm exploring now for personal work is to have a Lisp(or something that comes close enough in size, dynamic behavior, and convenience, like Lua) targeting a Forth. The Forth is there to be a stack machine with a REPL. You can extend the Forth upwards a little bit because it can metaprogram, or downwards to access the machine. It is better for development than a generic bytecode VM because it ships in a bootstrappable form, with everything you need for debugging and extension - it is there to be the layer that talks to the machine, as directly as possible, so nothing is hidden behind a specialized protocol. And you can use the Lisp to be the compiler, to add the rubber-stamping semantics, work through resource tracking issues, do garbage collection and complicated string parsing, and generate "dumb" Forth code where it's called for. That creates a nice mixture of legibility and configurability, where you can address the software in a nuts-and-bolts way or with "I want an algorithm generating this functionality".
> IMHO, a perfectly reasonable approach I'm exploring now for personal work is to have a Lisp(or something that comes close enough in size, dynamic behavior, and convenience, like Lua) targeting a Forth.
I've had this idea for a while now but never got around to actually executing it. I'd love to follow your progress if you're doing it publicly.
Because it's weird enough to be off-putting, and it doesn't solve a real-world problem.
If you look at the arguments in favor of Lisp, they'll often boil down to it being "beautiful", "elegant", "flexible", or even "magical". It's a very minimal language which allows you to do absolutely anything - a lot of which would be an absolute nightmare in most other languages. You could implement just about any programming paradigm in Lisp if you want to. I believe this makes it very appealing to computer scientists, or other people with a more mathematical background. However, this flexibility is also a massive footgun: if you're not careful your junior developer might end up reinventing the wheel a dozen times, and writing completely unmaintainable code in the process.
On the other hand, most other programming languages look kind-of the same. They are all quite opinionated about how stuff is supposed to work, with a lot of hardware details leaking into the language. However, they are very easy to learn: most of it is just "do a bunch of operations in succession" taken to the extreme. Anyone who knows C# will be able to pick up a basic understanding of C, Python, or JavaScript well within a day, and a lot of people new to programming will be able to write not-completely-terrible code within a month or two when given the right guidance. They don't need to know about all the abstractions and technical details to be a functioning member of your team.
When you're running a business, you don't care about any of that beauty or flexibility. You want code which is quick and easy to write, trivial to read, and understandable by even the worst programmer in your company. In practice that means in your comparison you'll be choosing Python over Lisp. Heck, Go was developed entirely around this principle, cutting out as many language features as possible. And because all the other companies are making the same choice there will also be way more libraries for Python, making the gap even larger.
So yeah, in a stroke of irony Lisp isn't more popular because it is better.
> Because it's weird enough to be off-putting, and it doesn't solve a real-world problem.
Nonsense. Quoth Wikipedia, "Lisp pioneered many ideas in computer science, including tree data structures, automatic storage management, dynamic typing, conditionals, higher-order functions, recursion, the self-hosting compiler, and the read–eval–print loop." These are all solutions to real world problems.
So many languages borrow features that were originally developed in Lisp.
> However, this flexibility is also a massive footgun: if you're not careful your junior developer might end up reinventing the wheel a dozen times, and writing completely unmaintainable code in the process.
Plenty of code written in Lisp looks just like your Python, Ruby, JavaScript, Java, et al programs where you're defining structures or classes, writing and calling functions, importing useful libraries, etc. Plenty of this Lisp code is just as maintainable as the non-S-expression code.
> Anyone who knows C# will be able to pick up a basic understanding of C, Python, or JavaScript well within a day
Same with Lisp. It's just:
(function arg1 arg2)
Instead of
function(arg1, arg2)
> You want code which is quick and easy to write, trivial to read, and understandable by even the worst programmer in your company.
There is plenty of "lowest common denominator" code like this written in Lisp. Much Lisp code is not buried under inscrutable macros, just like not all Java code is buried under layers of inscrutable classes.
That isn't the problem of learning Lisp. The problem with learning Lisp is
(`a ,b c)
and the infinite power thus infinite responsibility of the wizard programmer.
> There is plenty of "lowest common denominator" code like this written in Lisp. Much Lisp code is not buried under inscrutable macros, just like not all Java code is buried under layers of inscrutable classes.
But the problem remains: it only takes one wizard to make reading code impossible by outsiders.
The canonical way to write a Lisp system is to define a DSL (or, should I say, system-specific language) and implement the system in that. But this means no-one outside of the language/system developers know the language, this means Lisp tends to be write-only by design - not in the line-noise meaning, but in the obscure foreign language meaning.
You certainly can not do that, but if you choose to not do that, why pick Lisp? You can argue that all software systems develop their internal system-specific languages, and I agree - but the overall strict language provides a rigid framework for anchoring understanding, something you have to conciously work for with Lisp. (Note Java pre-2020 was so limited in expressivity it wasn't any better, because all the complexity ended up in the class hierarchies you reference - it was more similar to Lisp in that way than many realize.)
> One Java architect can make the code unreadable, even for himself.
I agree, even in my original post.
> But even those DSL language patterns can be learned. In many non-Lisp projects, the DSL gets implemented in C/C++.
The friction is everything, otherwise we'd all be programming Turing machines. In Lisp, it is zero, by design; it can be good or bad, as most things in engineering. Consequences of uncontrolled growth of a DSL vs imposing artificial process limitations for the future of the software project must be considered explicitly; this is automatic when the cost of starting a DSL is non-zero.
If you can understand "String ${interpolation}", you can understand list quasiquoting.
> But the problem remains: it only takes one wizard to make reading code impossible by outsiders.
This really is a Lisp meme. There are plenty of Lisp wizards like Guy Steele, Rich Hickey, and Matthew Flatt. The wizards perform the magical act of making code legible and intelligible. I have stumbled around several Clojure and Racket code bases and never felt like "I should understand this code but the features of Lisp make it impossible to know for sure." "Infinite power" macros and whatever are really only used sparingly and generally when it's impossible to achieve a goal otherwise. No one is doing (define + -).
> But this means no-one outside of the language/system developers know the language, this means Lisp tends to be write-only by design - not in the line-noise meaning, but in the obscure foreign language meaning.
I, as a Racket novice, have been able to add candlesticks [1] to the plot library without learning much about it. I have also debugged DrRacket (an IDE) to uncover that Racket GUI operations performed significantly worse if non-integer scaling was used [2]. At no point when I was going through Racket internal code did I ever feel it was write-only. In fact, it was quite convenient to modify Racket internal source code, rebuild, and test changes in a way that would be much more difficult in Java or C++.
> You certainly can not do that, but if you choose to not do that, why pick Lisp?
Built in rationals.
The ergonomics of defining [XML / JSON / etc] data as S-expressions and doing things like pattern matching on that data.
Great, coherent integration between GUIs, plots, statistics functions, and all the other bits of Racket's batteries inclusions.
You still have access to all the other great features that other languages have borrowed from Lisp like REPL development, package managers (edit: maybe package managers were not a Lisp invention), good IDE tools, etc.
It is nice to learn the meta-syntax of parentheses once and know that the code will always look like that. No need to consider if some feature is implemented as a syntactically different new keyword, annotation, function call, or whatever. It'll always be a (feature).
> something you have to conciously work for with Lisp.
Plenty of languages have style guides, linters, static analysis tools, etc. to make sure the code conforms to certain restrictions. Lisp feels no different in this regard.
> When you're running a business, you don't care about any of that beauty or flexibility. You want code which is quick and easy to write, trivial to read, and understandable by even the worst programmer in your company.
This is exactly right, I don't know why so many people still fail to grasp this concept.
> Why do people write libraries for these languages and not the Lisp ones?
1. They want their names to be widely recognized, so they find a popular bandwagon to hop onto.
2. Raw numbers? More people using Python means more people trying to make libraries for Python, means more libraries remaining in the race after you eliminate the crap from people who don't know how to write libraries.
Note that Python is, by now, an old language. It wasn't instantly popular, and you wouldn't have predicted it. In, say, 1999, you had to be some GNU/Linux person to even know what Python is. It was far from obvious that, of all things, it would get so popular. That Eric Raymond article in the Linux Journal around that time probably gave it a bit of a boost.
Python definitely rode on the coattails of increasing GNU/Linux popularity, too. More people using Linux started asking questions how to script this and that, and going "gack!" at shell or perl programming. It seems Python might appeal to survivors of VisualBasic shifting gears into GNU/Linux stuff.
For the same reason people can't change their well-established habits and opinions, particularly those that have network effects.
Language popularity has little to do with how well languages are designed or how simple they actually are. Most popular languages are popular for historical reasons.
This is a great talk on exactly this topic from a Clojure conference, if you want a long answer. It focuses on the “functional” feature of Clojure but all of it applies to the whole language (and probably any good lisp) IMO. https://youtu.be/QyJZzq0v7Z4?si=y3hhYaInMkRLBCJK
My brief text answer (focused on Clojure):
-python has been around about 20 more years than Clojure
-the advantages of Clojure vs python/etc probably aren’t nearly as big as python vs a non memory managed, compiled, static language like C. That whole generation of “scripting” languages had the wind at their backs in a way Clojure and its contemporaries never quite will (though Clojure et al tend to be much better at concurrency and parallelism and this will help a lot)
-unfamiliar syntax (not Algol like) and paradigm (not oop)- and the truth is many programmers back away slowly when a thing is too alien
-hasn’t found a niche as big as data science or scientific computing or CRUD website building - I think python has aggregated some great academic niches, at least one of which (ML/data science) exploded in popularity. Ruby had the rails community. Clojure seems to have some popularity in fintech but has no big single niche yet that it dominates.
>If it's because of libraries, then the question is: Why do people write libraries for these languages and not the Lisp ones?
Because libraries already exist for those languages, as well as support, vendors, and a big ecosystem, familiar syntax, and jobs. So they write libraries for languages that are already popular.
If you meant, "but why wasn't some Lisp the one that gain popularity back in the day, when C, C++, Python, and Java didn't exist or where still fresh?"
I think because:
1) it was too advanced for the procedural mindset at the time,
2) it was not sufficiently efficient in those primitive 16 bit machines
3) fragmentation
and most importantly, no killer app and major vendor backing or OS first-class support (like C had for UNIX, C++ for Windows, and Java got from SUN).
I don't think anyone is saying Clojure is better than Python in the abstract. Clojure has a different style from Python and that style can be more fun and is better for some specific tasks like application programming and anything that wants to use multiple CPU cores. Tends to have better long term prospects too, old Python code doesn't work in my experience but things on the JVM have cockroach powers.
As for why things are and aren't popular, who knows? It is quite possible that people just don't like the look of the parenthesis, or there are a couple of key libraries that aren't good but nobody vocal has put there finger on which ones. The error messaging isn't quite up to a standard that people want to deal with. Or maybe popularity is really just about pure random chance, at most 4 languages get to have >20% market share, by definition.
Because of habit. There is always a cost in trying to learn something new, to the level you are on with some other language and it's tooling and libraries.
Programmers who program in some language for a few years realize they are still learning more about it, all the time. Therefore they know if they switched to another language it would also take them a few years to attain the same level of mastery.
And then there is similar inertia with an organization that has lots of code in some specific language already.
For some people, community signals are very important. Massive conferences or raw number of libraries, etc., indicate some inner quality of that language's ecosystem that they value.
Social forces and economics. Lisps were winning in academia for a while. Unix started winning in the engineering world before Lisp systems really got their feet under them. Engineers made a lot of money, and academia shifted focus to the systems that were making money. Now everyone programs for Unix instead of Lisp.
Most Python users aren't software engineers. They're students, scientists, business analysts... We're lucky that we were able to drag them away from Excel. Asking them to learn yet another programming language might be a bit much. And if they did, it wouldn't probably be a lisp (it would probably be among: C, Javascript, Julia, Go, Nim).
I want to move on from Python, but there are so many more people that I can help if I stay.
I have the same problem. It is a good language to teach to non-cs majors. If you want to build something out of the box, use Python. However, GPT is coming and maybe we won't even use Python anymore for simpler tasks.
At the end of the day, things get popular through social processes, so it's far more a matter of social factors—some of which are practically random—than any intrinsic qualities of the thing itself.
