To really discover what interactive, image-based CL development is about, my pointers:
- https://www.youtube.com/watch?v=jBBS4FeY7XM (5min) my demo to show how to resume execution of a long running program, from the stack frame right before the bug, after having fixed and recompiled the function, without quitting the debugger hence without stopping the program, thus without restarting everything from zero.
- and no: while we could deploy a ball of mud to production, we can also not and use traditional tools and techniques: build an executable from scratch, restart the app, reload state. But we can do everything in between too (inspect the running app, change it a bit, change it a whole lot (while staying in sync with local source code), use images to speed-up development etc).
To a point. There's an entire culture of software and processes around building up container images from scratch for deployment. There's a term of art that I forget, "Pets vs XXX", with XXX being disposable VMs, in contrast to a Pet computer lovingly cared for over time.
For many Common Lisp scenarios, it's actually similar -- start with the raw based image, load in your code and data, snapshot it, and ship the resulting artifact. At least for me, interactive development aside, it's not untoward to restart a CL system from scratch as a routine part of development. We're not running on Lisp Machines anymore where restarting was actually quite expensive (though there were anecdotes about how at some point it was faster to restart a LispM than let it grind through its GC process, I don't know about that).
However, I don't know how common that is with Smalltalk. How often they restart from scratch. Saving the image is routine in Smalltalk. Where in Lisp you CAN save the image, it's not necessarily part and parcel operationally, historically, or culturally to constantly save the image, especially post LispM. In Smalltalk it is.
I think with CL folk moreso build a new base image, load in some routine utilities, etc. to form a baseline. But application code and image are separate.
Smalltalk, they're not. Code is part of the image (it's stored separately, but it's presented as if they're one and the same). So the image is a much more first class citizen in the Smalltalk world, rather than, perhaps, an artifact.
Especially today, I imagine they have build processes that build up the entire thing again from scratch, but it would not surprise me if they bumped into problems where the image and the source code "don't quite match" that they have to suss out in testing before deployment, because something slipped through and tweaked the long saved image that wasn't properly captured.
In any case, while I'm sure it doesn't quite happen like that today, its not unreasonable to have someone ship a "ball of mud" Smalltalk image in contrast to one in CL.
Generally Common Lisp is agnostic to the development style. For production, there are also batch compiler. Image saving/starting is also not supported by all implementations (not supported for example in ECL and ABCL).
In early Lisp I would guess it was more often to save an image. There is one system which works mostly like Smalltalk: Interlisp-D, now called Medley. https://interlisp.org It also manages the code, in some ways similar to Smalltalk. Interlisp was provided by Xerox PARC, like Smalltalk. It originates from BBN Lisp, from the 60s.
Other than that, most Lisp systems store code in the file system.
> its not unreasonable to have someone ship a "ball of mud"
That's also relatively common (!) in Lisp.
In Smalltalk we often see variants which can ship images and run them on different architectures. In Common Lisp images are mostly tied to a certain architecture & OS.
> But we can do everything in between too (inspect the running app, change it a bit, change it a whole lot (while staying in sync with local source code), use images to speed-up development etc).
Is anyone aware of SBCL or LispWorks-based tooling that integrates with ServiceNow CHG records to record and perform these kinds of changes to automatically create an audit trail of such changes in a CHG record as the Lisp image is manipulated for emergency changes?
After getting sbcl up with visual studio code and the alive extension, my mind was totally blown,
I'm certain that emacs/slime is the true pure way, but for someone who's not wanting to learn emacs, alive is just pretty amazing and magic, and it fits well with what is described in the article. I wish I could do stuff like recompile functions in-place and just retry the call without exiting the process in every language in this way.
The IDE situation remains the #1 issue in adoption. Emacs+SLIME+Paredit (or variants) really are the best way to write Common Lisp, but it's also extremely unpalatable to someone who isn't willing to invest in not just a new language, but also an entirely new and extremely intricate editor stack.
It's been my hope for well over a decade that the situation would change. Alive is decent, but has a long way to go before being a tool a professional could reasonably use.
> The IDE situation remains the #1 issue in adoption.
Im curious why you think this is the no.1 issue? I think a bigger issue is that so many CL programmers prefer to talk about lisp instead of just getting their hands dirty and making cool original open source things with it that other people would get excited to use. Thats actually a much harder problem. Plenty of people actually use Emacs even if they dont care for elisp and end up learning and writing elisp because they like Emacs. I think CL should market itself to uni or even high school students. Kudos to you though! You are doing great work to promote CL
Honestly this unreasonably scares people off. This apparently very large loo loo of a first step, setting up all this infrastructure just to get started.
The tooling is nice, but it’s not what defines CL. The language has ample benefits just on its own.
A paren matching editor is the minimum tooling necessary for CL. Auto indent is a nice to have. vi has had both for pushing 50 years now.
The REPL is a standout feature of the system. Being able to load your code and poke at it is wonderful. Very powerful feature.
Python has a REPL. I never felt it was very good, but that’s me. I never used it much. Perhaps I was missing something. I wrote a bunch of Python without ever using it. And I’d wager many coders have treated Python like they’d have treated Perl, or most any scripting language. Write code, save code, run script, rinse, repeat.
With no interim step, turn around was fast and work got done.
There is nothing stopping folks from doing the same with CL. I’ve developed 1000s of lines of CL with little more than 2 little defuns. (e) and (l). (e) fired off ‘vi code.lisp’ and (l) wrapped ‘(load “code.lisp”)’. My knowledge of the debugger was knowing how to abort it back to the top level. I mostly did “print” debugging. If I was clever, I’d write a ‘dprint’ function that would take variable arguments. If I was lazy, I’d just “(print ‘(“x=“ x “y=“ y))”. No big deal.
Even when working with LispWorks (which has much to recommend it), I operated much like this. Reload all the code, dabble with the REPL, add print calls, reload, test individual functions. I never took the time to learn the higher level tooling and such. I didn’t feel it was necessary, and I’d rather code than fight tooling.
So, as the Lisp cognoscenti stares on in mouth agape horror as I left much of the Lisp experience on the shelf, I got stuff done anyway.
Which goes back to the fundamental point. CL is a language like any other. It’s a very cool language, very powerful, and worth using just for itself. Its environment does not define it as a language. It augments it, it’s cultural and useful to be sure, but you don’t necessarily need it to be capable and productive and to solve problems.
If all the talk of tooling and what not deters you from trying the language, don’t let it. It’s worth trying all on its own. Try it your way, however you’re comfortable.
But do get a parenthesis matching editor.
Edit: just to highlight the point, if you read PGs “On Lisp” and “Common Lisp” books, you’ll notice they’re pretty much all about the language and its power. He does not talk much as I recall about the environment.
Nice! I'll note, specially for pythonistas, that even with your simple (edit) and (load) shortcuts, you are doing something more than in Python: while in Python you quit and re-run your program from the terminal, in CL you quit the debugger, you stay in the running Lisp REPL, you edit code while your REPL still around, and you re-load code from the REPL. It's already a faster write-run loop, and you can add advantages like keeping all your test variables around.
As a huge Lisp/Scheme fan, I don't find it surprising that Python became the language of choice for neural networks over Lisp.
NN in general just boil down to doing lots of linear algebra, which is a lot about mutating very large matrices. For this you really just want a wrapper around BLAS/LAPACK, so you can leverage existing optimized libraries.
Working with matrices in Lisp always feels a bit clunky, and doubly so when you want to do a lot of stateful operations on them. The real compitetor in this space to Python is something like Matlab, which has probably the best interface for doing linear algebra (but is worse at everything else).
The one area where Lisp and Python both shine is the ability to perform automatic differentiation. Lisps are great for this task since it's all symbolic manipulation. However, this is only important once you have a solid interface for working with matrices.
If you want to get a feel for the difference I would suggest reading through the surprisingly excellent The Little Learner. I think you'll find that while it really demonstrates the power of Scheme (Racket in this case) in areas where it excels, you wouldn't want to use the framework in that book for anything other than toy examples.
> But is there is something inherent to Lisp that makes it "clunky" to work with matrices?
s-exps are basically coding at the level of the AST, which for many programming tasks is a powerful level of both abstraction and control.
But this interface tends to work much better on tree-like structures (the most simple of which is the list, which in it's simplest case is a cons cell, Lisp's most fundamental particle).
Most programming tasks can boil down to tree manipulation, but large matrices feel a bit out of place with this interface because you are basically to working with arrays (which in it's simplest cases is a pointer to a memory address, which is C's fundamental particle). C like languages (I know, technical ALGO-like), tend to feel more natural for these tasks.
But of course if you can manipulate trees easily, you can manipulate code easily which gives you...
> Lisp can be at least as high level and expressive as any other language.
In practice, for Lisp this is also a weakness. Again, I love Lisps in general, and of course any Lisp and especially Common Lisp (with it's exceptionally excellent macro system) can do anything you want.
However, my (and many others) experience has been that this leads to is it becomes very easy to create code that is amazing for the original developer, but very difficult for new developers to get their head around. Hence the adage "Lisp is optimal for team sizes of one."
You could undoubtable build the worlds most elegant interface to working with matrices in Lisp, but now you essentially have a new language.
I think matlab has the most succint language for writing matricies
[1 2 3; 4 5 6; 7 8 9]
then lisp
#2A((1 2 3) (3 4 5) (7 8 9))
then python comma freak show
[[1,2,3],[4,5,6],[7,8,9]]
python's greatest achievement was bringing open source to matrix calculations and dethroning matlab. python is what it is today because it offered 99% of what matlab offered in open source and for $0 to uni students
I was curious about how Julia does 2D matrices and arrays, given they target mathemeticians. They seem to have no shortage of methods of creating them.
octave was always seen as subpar version of matlab and r is a statistics programming language. in comparison python was already back then a proper practical programming language that people were using for things other than math
Lisps have n-dimensional arrays since decades. Often their implementations are a bit less efficient than the ones found written in C and used by Python.
> You could undoubtable build the worlds most elegant interface to working with matrices in Lisp, but now you essentially have a new language.
My understanding is that enough time had passed between the AI winter of the late 1980s and the rise of big data and deep neural networks in the 2010s to where a relatively tiny minority of modern machine learning practitioners came from the old-school AI community, which tended to prefer Lisp or Prolog. When machine learning became hot in the past 15 years or so, many new entrants to ML who had no previous AI knowledge used the tools they’re familiar with, which are generally C, C++, Java, JavaScript, and Python. In addition, many in the natural sciences use Fortran and Matlab. Perhaps had the AI winter of the late 1980s not happened (which took out the Lisp machine market) and had Common Lisp not been relegated to certain niches, perhaps there would’ve been more usage of the language, which may have led to Python being less dominant today.
I love Lisp and Smalltalk, but I work as a machine learning researcher as my day job, and I code in Python at work. Admittedly I miss Common Lisp whenever I code in Python, and I think Jupyter notebooks are a poor man’s Smalltalk environment, but when comparing Python to C and C++ (and I used to work primarily in systems programming, so I know my way around C), Python is a major productivity boost.
well, if we categorize python and js as non-lisp languages, i'd say it's mostly because python has been the language of choice at google since last millennium, at least for situations where flexibility outweighs performance
last millennium when google started up, no lisp was a good option. clojure didn't exist, schemes were mostly for teaching, cmucl was in disrepair, and though there were proprietary common lisps, they were proprietary and wanted per-cpu license fees, which would have meant disclosing how many cpus google had—a closely guarded secret
moreover, common lisp's approach of putting a uniform but somewhat repulsive veneer over all operating systems was a pure drawback in google's all-linux environment, and s-expression syntax is harder to read than python's (if easier to edit)
also the python community was friendly and welcoming, while the scheme community was tiny and fragmented, and the common lisp community had degenerated into an ego trip for a world-class asshole named erik naggum
this is guessing from the outside; i've never worked at google, which is why i'm allowed to write this
I like that you're framing this as if it was between python and lisp and not python being the language of choice by a huge margin followed by a dozen other languages, then lisp.
Maybe when Common LISP gets its next standard and stops being a geezer on the deathbed ? The world has changed since 1994. So has the computing landscape. If a programming language is static for more than a human generation, then it cannot expect any sort of adoption.
It's unlikely the ANSI standard will ever get updated - it's a very expensive process.
Python doesn't have a standard, but it has a canonical implementation, so maybe a "defacto standard". Not having an official standard didn't matter, though.
Common Lisp has defacto "standard libraries" for things like threading. Other things, such as async, are less clear (to me), but libraries do exist. Just because something isn't in the standard, though, doesn't mean it can't exist. Also, implementation such as SBCL pick up a lot of the slack, filling in some of the missing gaps.
Python evolves through PEP's https://peps.python.org. Can you kindly correct my deplorable ignorance and point me to the Common LISP enhancement proposals ? Maybe I am missing something elementary.
And? What has changed so much since 2013 that would require new proposals? Do you need the language to change every month? You keep moving goalposts, and I'm beginning to think that you’re not arguing in good faith here.
Not sure what you mean by arguing in good faith as its a straightforward argument.
So much stuff has changed since 2013. Containerization, massive increase in concurrency, mobile platform explosion, etc. Where is the concurrency standard ? Are these 14 CDR's even implemented by CL implementations ? Can't find which CL implementation implements priority queue. Several of those CDR's don't even appear active.
Really ? We are simply going to ignore the entire Python web-app landscape ? Django, Flask, Pyramid, etc. The deployments of those outnumber CL deployments by so many several orders of magnitude. No, ML didn't "save" Python. It only accelerated its usage further above all competitors.
Django and other web apps pale in comparison to Python's dominance in numerical computing. Im not saying they are not important, in fact I believe Python's use in areas other than math drove its adoption in the scientific community, and that one growth fed the other, but without ML there are far too many competitors to Python. However, almost everyone agrees thag py2->py3 was a mess. I like that when I run CL code written 10/20 years ago, and even older!, I dont have to worry too much if it is going to run today
Not sure what this for and what is it to do with interactive part of development. If you are vim people try search for vim and lisp and got there. emacs people obviously no need.
For someone who knows Common Lisp but rarely plays with the redefinitions mentioned in the post, it's quite enlightening to learn those things.
If you're not into Common Lisp you're definitely not the target audience for a post looking at the weird details of it like this one.
I am into it. And he has some more info. Just not here or there.
For batch mode you can run script. Using format print :-). But in real life when you encounter problem the great thing of lisp is you can curl-c and inspect what is going on. I just doing my study of cl-6502 and found that this interactive nature and the readable of lisp is invaluable.
For interactive mode you can do rely upon emacs (or ccl which I cannot use for look like a decade now due to sudden issue; finally resolve this using xaddr) or vim (google vim lisp lead you there) where you can load an image and then interactive with it.
Why it stop here in my script? Just inspect around.
Now this is somewhere possibly this article in the interactive mode to see how symbols, … etc might be useful. The problem I have (I just read it the third times trying my best to appreciate) is that it does not lead you to anywhere. If you do not know these the article does not help. If you do, it does not help much as it is not going deep enough.
Perhaps an interactive session or information how interaction and use of this concept in a future article might be useful. By this article alone more is expected when one read that.
Just in case I am definitely no better than him on lisp. I just expect to stand on another “giant” shoulder to learn more about this lisp approach. Sorry if I think it shortfalls a lot.
It is about the little things that might bite you during a long development session, like deleting a generic function in source but still having the function in the image, or the strategy to update existing object instances when a class definition is updated (and slots are modified, added or removed).
---
btw, regarding packages and removing a symbol: uiop:define-package fixes the annoyance.
- https://www.youtube.com/watch?v=jBBS4FeY7XM (5min) my demo to show how to resume execution of a long running program, from the stack frame right before the bug, after having fixed and recompiled the function, without quitting the debugger hence without stopping the program, thus without restarting everything from zero.
- https://www.youtube.com/watch?v=nsKx40ab9SY (20min)showing what saving and restoring an image does, differences between SBCL and LispWorks.
- https://www.youtube.com/watch?v=1aboMZkNDCY developing a 3D graphics software.
- https://mikelevins.github.io/posts/2020-12-18-repl-driven/
- debugging CL tools: https://lispcookbook.github.io/cl-cookbook/debugging.html
- and no: while we could deploy a ball of mud to production, we can also not and use traditional tools and techniques: build an executable from scratch, restart the app, reload state. But we can do everything in between too (inspect the running app, change it a bit, change it a whole lot (while staying in sync with local source code), use images to speed-up development etc).