This is a very nice web site describing a programming language with unparalleled expressiveness, power and permanence.
I am heavily invested in Common Lisp. We are developing a programming environment for designing new materials and molecules called Cando (https://github.com/drmeister/cando) using Common Lisp as a scripting language. Cando is running on Clasp (https://github.com/clasp-developers/clasp), a new Common Lisp implementation that interoperates with C++ and is based on LLVM.
What attracted me to the language, after 35 years programming in almost everything else, was how organically it lets one write software and how I don't have to worry about it fading like the next programming fad.
Have you had an opportunity to look at Julia? It seems that there is some overlap in what can be achieved - with some obvious differences: as I understand it you had a lot of c++ - that could probably (today) be linked from julia, but not "integrated", starting from a green field with Julia, ideally one could do most things in Julia - maybe with some help from rust. And Julia is not a common lisp, obviously.
But in merging a high level language and the love, it seems Julia has been successful in getting a lot of real-world, "all Julia" libraries, with "sufficient" performance.
If you were starting over today - would you still prefer to build a common lisp with close integration to c++?
I did choose Common Lisp over Julia - I started learning Common Lisp in 2013 and Julia started becoming something around 2012.
I did so because of the permanence and the demonstrated expressiveness and power of Common Lisp. Common Lisp has been around for almost five times longer than Julia, and Common Lisp has demonstrated again and again that it is capable of solving hard, poorly understood, real world problems in a lot of domains.
Fun fact: Julia bootstraps off of an implementation of Lisp. Furthermore, most compiled languages are translated into an abstract syntax tree (AST) as part of compilation. Lisp S-expressions are a text based AST. So I can make the argument that Lisp is as close the "One True Programming Language" that we have. :-)
I'm doing chemistry - that's fundamental and timeless - I find it maps beautifully to a fundamental and timeless language like Lisp.
Since you're already using LLVM, I'm wondering if you've thought about adding something similar to the includec function in Terra [1] which uses Clang to parse C headers. I used to think that Lisp's CFFI was the best FFI out there until I saw that it was possible to literally parse a C header file and have it "just work". It seems like if you're already building against LLVM, so bundling Clang with your project too might not be that big of a jump for you.
We exposed the AST and ASTMatcher libraries of Clang inside of Clasp and we use it to analyze all of the Clasp C++ code to build an interface to the Memory Pool System memory manager.
I want to have something that can include C and C++ header files and automatically expose them to the Common Lisp - that would be neat!
Maybe we can steal some ideas like the includec function of Terra.
No, none, zero, zip, nadda. What you do is you hook it to another really important idea and then you work until 3:00am every night for months and then years writing code. You take all those voices in your head that say you should not be doing this and squeeze them hard until they... go to sleep and stop bothering you. For me it started paying off in the last year and it is starting to look like it was a really good idea.
And you're my hero. For having the big courage to write a completely new CL implementation, and attempting what wasn't attempted before: LLVM output, great C++ interoperability. Even more, leveraging the latest state-of-the-art compiler modules (Cleavir, etc.)
> how I don't have to worry about it fading like the next programming fad.
I'm not sure I understand this bit. What other programming language has "faded" and how has it "faded"? At least since the more mature age of software, which I'd say started around '95 or so (so 20+ years).
I say this with a perspective of watching it happen over forty years. I'll get all kinds of heck if I start saying this or that language has faded. But many languages have changed over time, sometimes in breaking ways, as their developers work to make them more expressive and add features. Code that is written one year often doesn't work a few years later - that's a kind of fading as specific reference implementations fade away and code written in them needs to be updated or it rots.
Common Lisp doesn't have this problem - and it has many features that other languages have added over time, often in less comprehensive ways.
For example C++11 lambdas are not as expressive and powerful as Common Lisp first class functions, and I joke that C++ template programming is to Common Lisp macros what IRS tax forms are to poetry. Note: Clasp includes a lot of C++ template programming and C++ lambda code that I enjoyed writing and I am so grateful that C++11 added variadic templates.
New language features can be added to Common Lisp by the programmer with macros, and for the really adventurous, they can be made efficient by augmenting the Common Lisp compiler. Common Lisp compilers are almost always written in Common Lisp and available and accessible. This is why Clasp uses a new and understandable Common Lisp compiler "Cleavir" (developed by Robert Strandh) and LLVM, which is a great C++ library for generating native code on a variety of processors. Cleavir will be an excellent platform for efficiently implementing new dynamic programming language features.
Thing is, except for CS theory evolving, the programming language environment from 30 years ago is in my opinion irrelevant from a maturity perspective. The number of practitioners was probably 1 million or so worldwide, now there’s probably 20-40+ million programmers worldwide. We didn’t have widespread internet access, barely a handful of Open Source communities, compilers were generally commercial and extremely expensive, companies behind technologies were extremely volatile, etc.
While that made for very nice war stories, it also meant that you’d have the “ship” sinking under you completely. Now a tech still stays around to some degree.
A C developer from 1990 could still be a C dev in 2018. A Java dev from 1998 could still be a C dev in 2018. And a programmer in your average mainstream language in 2018 could reasonably expect to be working in the same language in 2018. Especially if that language has enterprise traction (Cobol, Java, C#, Javascript, SQL, etc.).
Also, languages are a lot more similar to each other than they used to be, in terms of expressiveness and power. Lisp was a “lightsaber” to the “stones” of the 80’s, now it’s at most a slightly more powerful “machine-gun”, at best.
And regarding language evolution itself, everything evolves. Or did Common Lisp programmers generally write unit tests, have integration test harnesses, use package repos, etc. back in 1995? I somehow doubt that :)
Common Lisp has: generic functions, homo-iconic macros written in Common Lisp, reflection, multiple dispatch, closures, optional typing, correct implementation of lexical scope, dynamically scoped variables, a metaobject protocol, conditions & restarts, and a compiler that is available for customization. One or a few of these have made it into other languages in some form or another - but not all together.
Common Lisp also has an excellent package manager (quicklisp/ASDF) a programmers IDE (emacs/slime vim/vlime), interoperation with C libraries (CFFI) and lots and lots of libraries that don't need to be constantly updated to keep them working because the underlying language changes.
I wouldn't call it just a slightly more powerful weapon.
Just the macros - my goodness - in Common Lisp you can write programs that write programs! It's easy and organic and elegant because they are written in the same language as everything else. Macros let you write code that customizes itself and optimizes itself at compile time for specific use cases. You can escape the drudgery of writing similar code over and over again. You kind of need s-expression syntax for macros to be easy to write - so no other language with complex syntax will ever have them like Common Lisp does.
Languages are all Turing complete - we can do everything in every language - but for the kind of hard problems that I want to solve, only Common Lisp has helped me out of the Turing tarpit. Also, C++ is a great domain specific language for creating tables of cache friendly compact data structures and it has great compilers.
As a Professor of Chemistry - I'm all about higher learning and I love curves (non-linear functions that is).
I don't agree that Common Lisp has a higher learning curve than other languages. To become operational - Common Lisp is no more difficult to learn than other languages. Bind variables, conditionals, loops and a little arithmetic and you can start doing some damage. You can write procedural code, functional code, object oriented code - whatever your problem needs.
It's that Common Lisp has a LOT more "up" than other languages. It keeps going up and up as you learn about macros and CLOS and multiple dispatch and restarts and upwards. It's really neat discovering all the things that you realize you were missing from other languages - some things that they add over time with ever more complex and arbitrary syntax.
When you get tired of writing "for" loops and agonizing how to manage memory for the next fancy data structure you want to implement - dive into it - it's great. Common Lisp will still be there.
>>At least optional typing and macros are not considered to be universal benefits for programming.
They are, if they use them enough.
Some body mentioned in this post that number of programmers from the 1980s has gone up from probably a million to 20 million or so.
What happened is that we had to lower the bar to entry, such that we now have many novices and a few experts. Most people want to just come in coast 9 - 5 and go home, so you have very few people who want to push the limits of their tools.
You seem to be a bit too... enthusiastic about this conversation, for my tastes :)
I won't go into everything cause it would take way too much time, but I have to point one thing out:
> Optional typing dramatically increases execution speed, something that is very relevant and important in 2018.
As compared to what? I'm guessing your assumption was that I was thinking about purely dynamic languages? Not really, I was thinking about using static languages with type inference.
I don't understand what point is. Do you agree that Lisp metaprogramming has benefits or do you disagree with that?
If you agree, then what is your point? That we should not use a powerful language because it is not popular? Or that features in popular languages are the only features worth having?
If you disagree, can you elaborate what exactly about Lisp metaprogramming do you not find powerful? That would make your arguments more concrete. Otherwise you are just going around in circles stating the same opinion over and over again which is really only your opinion that many of us don't seem to agree with.
True, but they support entirely different philosophies to software development: the "build a cathedral" (Haskell) versus "create a living organism" (Lisp), to paraphrase an Alan Perlis quote.
If one used a Lisp system at MIT in the 80s, probably.
The typical Lisp Machine at MIT was networked and shared file servers (also from other types of machines). When one uses a Lisp Machine on the network, one uses a name server (not DNS or NIS, which did not exist at that time) which define one or more sites. The site information in the name server provides information about machines, services, users, printers etc. In the 80s the MIT network was already vast with hundreds of machines. It was basically similar to what Unix system did with mounted NFS volumes, but here built into the Lisp system. This was before TCP/IP, when Lisp Machines used the CHAOS protocol for networking.
Now when one wanted to load some software into a Lisp Machine, one uses a command like 'Load System FOO', where FOO is the name of the system. The Lisp Machine shares with other machines a central registry for system names, which then point to a location in the file system. So this would find the system definition, load it and then load the system components. One could also ask it to load a specific version or load a specific patch level - or just the newest version. A system could also have other files like documentation, C files, etc.. There was also a way to write those to tape or an archive format and transfer it that way.
Thus that was a networked repository for libraries and applications. Some of those libraries were shared across different machine types and Lisp dialects. For example the LOOP feature was a library sharing source code with Maclisp, Zetalisp, NIL, etc. So you could load this from a central location into different Lisp systems - not just Lisp Machines.
"Or did Common Lisp programmers generally write unit tests, have integration test harnesses, use package repos, etc. back in 1995? I somehow doubt that :)"
Richard Waters wrote and made available a nice pair of utilities that I've long used.
The first is the RT package, which allows definition and execution of unit tests. Many elaborations of this have been written over the years.
The second, COVER, is more impressive: it uses Common Lisp macrology to implement a code coverage tool. Combined with RT you can not only unit test your code, you can confirm the level of branch coverage your unit tests achieve.
(I've since hacked up COVER to enable various extensions, including saving of coverage information for aggregation from different executions, and rollback to enable automated search for minimal coverage-improving inputs. Not available publicly right now, unfortunately.)
>Also, languages are a lot more similar to each other than they used to be, in terms of expressiveness and power. Lisp was a “lightsaber” to the “stones” of the 80’s, now it’s at most a slightly more powerful “machine-gun”, at best.
How many features coming from Lisp, ML, Haskell and academic languages have made it into mainstream languages now? Many. How many were in mainstream languages in the 80’s? A few.
- NONE of those langs have the hindley-milner type system, type inference and typeclasses of Haskell.
- even Common Lisp's type system is more sophisticated than almost all langs on the list, allowing for union types.
- all of those langs only allow you to write the code that will execute at runtime. CL allows you to specify if a function is to be executed at read time, compile time, and run time.
- NONE has an object oriented system with multiple dispatch nor a meta-object system. Unlike Common Lisp.
- None have metaprogramming based on homoiconicity (like Common Lisp), so the only way of macros is to use a cumbersome AST->AST library, and even then, it can only be done at compile time nor runtime.
- Only a very few allow changing the definiton of a class or function at runtime, without stopping the running program, and then it's often a difficult proposition because the language wasn't designed to do this on the first place. Unlike in Common Lisp.
- For the garbage collected languages there, NONE allows you to circumvent or disable the garbage collector, unlike CL.
- For the garbage collected languages there, NONE is faster than CL, and only CL allows the programmer to write Lisp code that reaches C speed, if he/she wishes so.
- only a few have built-in support for all of these: integers, arbitrary precision numbers, rationals, floats with IEEE compliance, complex numbers, and bit vectors. All which CL brings you by default.
Now, you can suggest more languages and I can go on.
There are very few languages that can compete with Coq, Agda, Haskell, Ada/Spark, Common Lisp, Julia and Racket today.
No, there are many features missing. Features that are very relevant. Do you know that many of the design patterns often used in Java are unnecessary in Common Lisp simply because many of them are workarounds for things that the Java OOP system lacks but Lisp's OOP system implements?
I do like Lisp, but you should learn better the languages you criticize.
C++ allows for compile time execution, improving since C++11. A feature also shared by Nim, D and Jai.
C# does offer control how the GC operates, and allows for manually memory management if really required. A feature also supported by D, Modula-3, Active Oberon.
.NET Expression trees can be manipulated at runtime.
There are GC enabled system programming languages that are as fast as Lisp, like D, Modula-3 or even the new .NET Native toolchain.
It’s simple math, really. Let’s say we had 1 million programmers in the 80’s, writing enteprise software costing 1m * 8h * 365d * 10y = too lazy to do the math, let’s say 30 billion dollars. A lot of that software died with the companies using it, so maybe 10 billions’ worth survived, needing today N C programmers, P Cobol programmers, etc. for maintenance.
How do you think the numbers and the timespans look from a time when we have 20-40+ million programmers, working with better tools on system which are even more tightly coupled with core business functions?
I didn't want to call out an obvious typo - but then I thought "maybe they are talking about Javascript frameworks" and then I thought "yeah - one can always hope". :-)
C++ template programming necessarily followed a very different style than it does since variadic template were introduced with C++11. There were books written before C++11 wherein much of them was obsoleted by variadic templates. You do not want to let students get hold of those books and think the old way was the way things have to be done. Compare how boost::python was (is? I haven't dug into it recently) written compared to pybind11 - night and day differences in readability thanks to variadic templates. That's not a dig at boost::python - it was written with the C++ they had back then and it had to use Lisp-like list comprehension with tedious replicated boilerplate code to unpack argument lists.
Then there are raw pointers in C++ - we aren't supposed to use raw pointers anymore. We are supposed to use references and std::unique_ptr<x> and std::shared_ptr<y> and this thing called "move semantics". That stuff sucks for directed graphs and so I switched to compacting, tracing garbage collection for Cando.
C++ is being developed in a way that it retains backward compatibility with existing code - thank goodness for that! But there are styles of programming even for something as staid and venerable as C++ that, while supported, should pass silently into the long, dark night.
Then you have a language like Python that made sensible but breaking changes between versions 2.x and 3.x. Wow - look at the problems that is still causing. I love Python but I won't do anything with it that I can't rip out root and branch with more than a few weeks work. I say that even though Clasp's build system is completely implemented using the very excellent Python based build system 'waf'.
Then you have Javascript - it seems to change every time I blink.
That doesn't mean I will stop using it for our web based Jupyter notebook user interface. Javascript is the only game in that town. I just close my eyes and hold on tight and try not to scream.
Common Lisp isn't like any of those, it grew out of a lot of very careful thinking and was specified around 1984 darn near perfect. There are some warts (logical pathnames - I'm looking at you) but even its warts are better thought out than some other languages features. Better implementations and more and better libraries come along - but the language has the same expressiveness, power and permanence it has always had.
Cobol? Seriously, there are probably more Cobol programmers than there are Python programmers.
And C? Most of the world runs on C, and its irreplaceable language in a non desktop computer.
And you will be surprised how much Perl code is written everyday.
The root comment is talking about the fad of the year, the kind of a language web devs use as flavor of the year. The current ones are Javascript frameworks. In this case the 'fade' is just the web framework, that probably has less mind share and open source dev contributions.
Yeah, when I wrote the comment, I kinda guessed that people would point out the zombie-like state of living for these languages. That's only barely being alive, though, and very much faded.
(I did have C in there but edited it out. I don't think that's a dead language yet.)
There are several implementations of Common Lisp that are still being actively developed.
Because Common Lisp is so easily extended, the ossification of the standard is actually a good thing. The stability of the standard means that code written 30 years ago will still work today. Feature updates aren't required, because any desired feature can be implemented in macros and imported as a library.
Well, yes :) And 4GL languages are actively being used to create new business, so who knows what's dead and what's not.
Perhaps the whole programming field is just a bunch of undead things. Sometimes a bunch of living things stumble upon the field, but they're soon infected...
The companies that use LISP don't talk about it much. I'm not saying secret weapon or anything either. I'm saying they seem to be among the set that just identifies something good for the business, buys/builds it, and solves business problems. They don't write articles about their programming language. What I will say is the small number of case studies on those two companies have much more interesting software than what's advertised for many stacks. The kinds of things people use it for corroborates the LISP advocates' claim it's a favorite for hardest, constantly-changing problems. It's more remarkable with Allegro since they charge royalties on top of the licensing with customers still buying their stuff.
>It's more remarkable with Allegro since they charge royalties on top of the licensing with customers still buying their stuff.
Do you mean that (apart from the licensing) they also charge a percentage of the revenue or profits that their customers make from products they develop using Allegro's products?
check out the new impressive built-in parallel features in Fortran 2018 here: https://goo.gl/ZbH4t7
GFortran 8.1 has already implemented the majority of it, and Intel is already adding the new features to their new Intel Fortran 2019. I don't know of any other mainstream language at the moment, that has native shared- and distributed-memory parallelism capabilities based on one-sided communications without recourse to the complex ugly out-of-language MPI library calls.
Perl, ML, Fortran, Cobol has not faded. You might not like them, you might not use them. But there are tons of them out there and being used in the real world and production systems.
I think it's fair to say those languages have "faded"
I would also say Python 2 has "faded" or started to.
Faded doesn't mean it isn't used anymore. It mean it's retreated into a niche, and is no longer on the short list for new development, and no longer interesting to most developers.
I am heavily invested in Common Lisp. We are developing a programming environment for designing new materials and molecules called Cando (https://github.com/drmeister/cando) using Common Lisp as a scripting language. Cando is running on Clasp (https://github.com/clasp-developers/clasp), a new Common Lisp implementation that interoperates with C++ and is based on LLVM.
What attracted me to the language, after 35 years programming in almost everything else, was how organically it lets one write software and how I don't have to worry about it fading like the next programming fad.