The Problem with Lisp 6 points by heinrichhartman 6 months ago | hide | past | web | favorite | 17 comments

 I don't understand the author.The evaluation order in Lisp is not "highly unpredictable", it is dependent on the form.I don't understand why he even constructs this strawman of "you must start in the innermost sub-form".That's not a rule that anyone ever made. That's the rule for arithmetic expressions, sure, but what does that have to do with Lisp?The rule in Lisp – as he later admits – is "look at the first symbol in the (outer) form".And the whole point of special forms is to make your own rules about evaluations. In order to help the programmer.Can you obfuscate the meaning? Hell, yes! Is that a reason not to allow it and to force obfuscated code by using unsuitable evaluation orders everywhere? Hell, no!And even in this bad example Lisp is certainly not more unpredictable than any other language.Let's take C:`````` int func(int a, int b) { return a + b; } `````` If you start at the innermost point "a + b" you get exactly the same questions: "Oh gosh, what is the + operator?", "How do I apply + to a number?" (really?) and "where do I store the result?"I think the author has misled himself into some extreme notion of the divine simplicity and purity of Lisp that nobody ever claimed to exist.
 Author here. Thanks for your comment. I appreciate your feedback!Let me first comment on your C example:> "a + b; // Oh gosh, what is the + operator?"There is no problem with the "+" operator. It's a language builtin. You can look that up in the manual.In the Lisp case, I did not pick on the "+" operator, at all. The problem is a line below:> "(sum x) ;; ??? Where does "sum" come from?"There is no function "sum" at all! It just looks like a function call, which is it not. You have to know what the enclosing special form "let" does to it's arguments.You could have those problems with C macros as well, but not in this example.> Can you obfuscate the meaning? Hell, yes!> Is that a reason not to allow it and to force obfuscated code by using unsuitable evaluation orders everywhere? Hell, no!I am not picking sides here. I am pinpointing a source of confusion. The Python/Ruby authors took another path and made the language less powerful, so they don't give developers the power to obfuscate the code in this way.
 Your article is confused between code walking (recognition of forms) and evaluation order. Code is generally walked top down for the purposes of expansion of macros, compilation or interpretation. Evaluation of nested expressions is generally bottom-up, and in well-behaved Lisp dialects, left-to-right.You cannot simply let your eyes randomly land on some (a b c) in Lisp code and assume that a is a function being called with arguments b c. You have to read the whole sentences and phrases, so to speak.No language is immune to this. Can you assume that * X is a pointer dereference? No, it could be a declarator. Or a fragment of a multiplicative expression. Or the interior of a comment as in /* X.(a, b) could be a comma expression, or the argument list of a function call.I can't think of any language in which I can extract any substring of a sentence that could be a syntactic unit on its own, and correctly understand its meaning without knowing the rest of the sentence.Lisp has to be scanned from the outside in: we have a defun, which encloses a let, and so on. When you're familiar with the piece of code you're working on, you can then take considerable shortcuts. You know you're in the middle of a function, even though the defun is off the screen. You know you're in the middle of the progn-like body of some construct (a loop, or let or whatever), so all the lines of code you see are evaluated forms.
 > Your article is confused between code walking (recognition of forms) and evaluation order.Yes, to some extend.> Can you assume that * X is a pointer dereference?[Aside: The C declaration syntax is particularly terrible. In a declaration 'star' means "address of", elsewhere it means "dereference". https://books.google.de/books?id=bXGhBAAAQBAJ&printsec=front...]> I can't think of any language in which I can extract any substring of a sentence that could be a syntactic unit on its own, and correctly understand its meaning without knowing the rest of the sentence.Well, most languages have expressions/statements that roughly correspond to source lines. When I read a line of python like this:`````` self.raw_requestline = self.rfile.readline(65537) `````` https://github.com/django/django/blob/master/django/core/ser...I can be 100% sure that:- `self` is an object with an attribute rfile- `=` is the assignment operator- there is a method `readline` that get's called with an integer argument 65537.All this without checking any enclosing socpes.A random line of lisp:`````` (setcdr current (list :args (cdr current))) `````` https://github.com/emacs-mirror/emacs/blob/master/lisp/wid-e...could mean anything depending on the enclosing scopes. Pretty sure setcdr,list,cdr are function calls but who knows. You don't have enough information to decide this from the snippet provided.
 > I can be 100% sure that: `self` is an object with an attribute rfile, `=` is the assignment operator, there is a method `readline` that get's called with an integer argument 65537.I think this is overstating things, eg, the class `self` belongs to might have defined __getattribute__, __setattr__, etc, so the things you know for 100% certain about that line of code in isolation don't really tell you anything about what it does (self does not definitely have that attribute, and that might not be assignment, at least); most of that you assume based on common sense, conventions, figuring the code is "normal", etc, the same as in Lisp. And that's ignoring a trivial case like if the previous line was `f('''`, where f may or may or not call eval.
 Reading this again, I realize that the variable name "sum" was chosen badly. I'll change that in the post.
 Neither Ruby nor Python support the idea of 'code is data'. Sure they are simpler. The concept of code is data is not about simplicity, it's there to support code translations.Most languages have constructs which don't behave like function calls and which have special evaluation rules. A simple example is the IF statement. It takes a condition and two expressions. Depending on the condition only one of the expressions will be evaluated. If you look around, you'll find a bunch of such constructs in most languages. Lisp also has such an IF operator. Lisp additionally allows us to write any amount of additional operators which have evaluation rules, which are not the evalation rule of a function call.Is the IF statement irregular? It isn't. It's just one of many operators which does not follow the evaluation rule of a function call.Something like Ruby has a lot of weirdness in its calling mechanism. Python has strange non-local transfers going on. There are lots of ways we can perceive 'irregularities'.
 > Neither Ruby nor Python support the idea of 'code is data'.Well, Python and Ruby have an "eval" statement. So "code as data" is supported, but discouraged. In contrast Lisp macros / "code as data" are commonly used.Why is there a difference in culture? Is the only reason, that strings are a lot harder to manipulate than lists? Is there more to it? Maybe translated code is harder to maintain and reason about? Maybe debugging and profiling is a lot harder (this is true for C macros at least).I have just shared one confusion that I was struggling with.I found python and ruby much easier to read and pick up. One reason for this is, that programmers are not allowed to define new irregularities. Maybe this insight is helpful for others, who are getting Lisp as well. At least this was my motivation for writing this post.
 > I found python and ruby much easier to read and pick up. One reason for this is, that programmers are not allowed to define new irregularities.The mechanisms they give you are more than enough, though. Ruby especially favours this kind of thing a lot; how many people really have any idea what's going on (and how many of them beyond "something to do with method_missing and instance_eval") when they write something like:`````` FactoryBot.define do factory :foo do bar { 'baz' } end end `````` I don't think understanding that it's all blocks and method calls wins you that much.
 Python and Ruby use flat strings as input to eval. That's an entirely different thing. To do ANY useful code transformations one needs to parse it. Lisp code already comes in a hierarchical token tree.Ruby and Python represent code as text. That's what Lisp does, too. One has a file of code. But Lisp has something which neither Python and Ruby do: the text is actually a data structure: lists, strings, numbers, vectors, symbols. Lisp provides a function READ.You can read code:`````` CL-USER 73 > (with-input-from-string (stream "(1 + 2)") (print (read stream)) (values)) (1 + 2) `````` It returns a list of three items.This then makes it easy to compute valid code and execute it:`````` CL-USER 74 > (with-input-from-string (stream "(1 + 2)") (let ((code (print (read stream)))) (rotatef (first code) (second code)) (print code) (print (eval code))) (values)) (1 + 2) (+ 1 2) 3 `````` Then put it into a macro:`````` CL-USER 75 > (defmacro infix (expression) (setf expression (copy-tree expression)) (rotatef (first expression) (second expression)) expression) INFIX CL-USER 76 > (infix (1 + 3)) 4 CL-USER 77 > (macroexpand '(infix (1 + 3))) (+ 1 3) T `````` Better, and complete infix macros exist.Or use infix conversion during reading. Write your code with sections of a more conventional syntax:`````` CL-USER 80 > (defun hypot (a b) "Compute the length of the hypotenuse of a right triangle with sides A and B." #I( sqrt(a^^2 + b^^2) )) HYPOT CL-USER 81 > (hypot 1 2) 2.236068 `````` Just put a quote in front of the expression and enter it at the repl -> the result is the transformed code.`````` CL-USER 84 > '#I( sqrt(a^^2 + b^^2) ) (SQRT (+ (EXPT A 2) (EXPT B 2))) `````` There is nothing like this in Ruby or Python.If you look at a Lisp interpreter, it actually executes Lisp data as code. Macros transform Lisp data as code. This is far far away from what Ruby or Python do. Python provides some access to its AST - that's also far away from what Lisp does.Lisp macros don't define irregularities. They do code transformations. Stuff where the language designer/implementor of, say, Python refuses to provide a control structure, you can do it yourself in Lisp in a few minutes or you can spend years to implement really cool stuff. The Common Lisp object system started as a large portable library (with only very little system dependent code), which code be loaded into Lisp and which the provided a very elaborate object system. This also included some clever macro code.I think titles like 'The problem of Lisp' may need a bit more reflection and patience. What is new for you is old news for the Lisp community. The exact same problem report can be already read in Lisp books from the 1960s. It's a bit like blogging 'The world isn't flat!.;-)
 > I think titles like 'The problem of Lisp' may need a bit more reflection and patience.Forgive me for the baity title ;) I realise this is not a new discovery.> Python and Ruby use flat strings as input to eval. That's an entirely different thing.I understand this. I was responding to the earlier assertion "Neither Ruby nor Python support the idea of 'code is data'." which is not entirely correct. You can clearly supply code as string data. (Python's namedtuple does make use of this). It's just not nearly as powerful.Thanks for providing the examples. The infix macro is pretty amazing! I will make use of this #I stuff.
 > Neither Ruby nor Python support the idea of 'code is data'." which is not entirely correct. You can clearly supply code as string dataThat's just some kind of eval feature. Many languages can execute strings of code at runtime - for example one could write C code to a tmp file, call the compiler, and load the object file -> that's how some Lisp-to-C compilers work. But it's just the usual C code as text.That's not 'code is data'. Code is data means that code is actually written in a data structure which is serialized and de-serialized. In Lisp code is written as s-expressions.'code is data' does not mean I can treat code as a string and evaluate it, it means: code is actually read, written and manipulated as structured data - and not as blob of text.Ruby code is written as textual Ruby code, where Lisp is written as Lisp programs in s-expressions.What you tell me is the code is text, which can be somehow executed at runtime, which is an entirely different thing.
 The evaluation order in ANSI Lisp is well-defined.In some Lisp-like languages such as Scheme, evaluation of function arguments is unspecified, similarly to C.
 It's a bit like claiming that riding a bicycle is hard because steering, moving forward and keeping balance at the same time is too difficult. Yet, people learn it and do it just fine. They just don't think consciously about it. Riding a bicycle takes a bit of training but after that the body movements are virtually effortless.Lisp is slightly more difficult than some other languages because lists/symbols are both used in code and data, because there are a bunch of different types of lists, and because code is a token tree.`````` (defun silly (a b c) (let ((my-sum (+ a b c))) (* my-sum (+ 1 my-sum)))) `````` Now what's wrong with that? Nothing. But people don't write that.Lisp is written like that:`````` (defun silly (a b c) (let ((my-sum (+ a b c))) (* my-sum (+ 1 my-sum)))) `````` There are a limited amount of tokens on a line. Syntactic constructs are not one line.Any definition has this pattern:`````` (DEFsomething name ...) `````` Oh, it begins with DEF, then it is a macro, followed by the name.Function definitions have this pattern:`````` (DEFun name arglist body) `````` The code layout is:`````` (defun name arglist body ...) `````` A method defintion:`````` (DEFmethod name arglist body ...) `````` The difference between a function and a method is in the arglist.LET is:`````` (LET binding-list body ...) `````` Now you need to know what a binding list is:`````` ((arg0 value0) (arg1 value1) ... (arg2 value3)) `````` Such a binding list is used in other places, too.If you give shortcuts to it:`````` df function definition bl binding list nm name bd body al arglist `````` DEFUN:`````` df nm al bd `````` LET`````` LET bl bd `````` PROG`````` prog bl bd `````` lambda`````` LAMBDA al bd `````` DESTRUCTURING-BIND`````` DESTRUCTURING-BIND al form bd `````` BLOCK`````` BLOCK nm bd `````` CATCH`````` CATCH nm bd `````` etc.Basically much of Lisp consists of a small number of basic patterns which are being combined. lists, assoc lists, property lists, arglists, binding lists, body, ...You've learned then a new alphabet of visual/structural patterns. Beyond that, code reading gets intuitive. Names give a clue what kind of pattern combination follows.The IDE can then help with syntax highlighting, syntax documentation display, etc.Lisp can even show you useful code layout:`````` CL-USER 68 > (let ((*PRINT-RIGHT-MARGIN* 30)) (pprint '(defun silly (a b c) (let ((my-sum (+ a b c))) (* my-sum (+ 1 my-sum)))))) (DEFUN SILLY (A B C) (LET ((MY-SUM (+ A B C))) (* MY-SUM (+ 1 MY-SUM))))``````
 Thanks for this elaboration!Obviously there is lot of community know-how, that I am not aware of. Thanks for explaining them in detail. Learning to spot all those pattern is crucial if you want to be come effective at reading lisp. For the untrained eye lisp code often looks horrid.I don't think we disagree, though. All I am saying is that: "Riding the Lisp bicycle has been difficult for me. Here one thing I struggled with a lot." Maybe other people have the same difficulty and find this helpful.
 Right, you need to train yourself with these visual/structural tree patterns. The code does no longer look horrid and the parentheses will go away.You were not knowing that there is actual structure and thus you could not see the forest through the trees.
 This is subjective. I came to Lisp with a background in C, C++ and numerous other things, none of them like Lisp.Lisp looked very good and highly readable; I liked it very much right away and was able to hit the ground running.

Search: