
Fun with Symbolic Derivatives in Lisp - taeric
http://taeric.github.io/CodeAsData.html
======
pflanze
Note that 'not requiring a parser' is only true iff you limit the expression
language you transform to one that has no syntactical constructs (i.e. can't
introduce new bindings). If you want to allow those then you need a "parser"
on top of s-expressions which maintains a bindings context (usually called a
code walker in the context of Lisp).

So not requiring a 'parser' in that sense is not really a feature of full
(modern) Lisp. (Early Lisps didn't use lexical scoping, and used f-expressions
or similar i.e. less of a compile- vs. run-time separation, so it may have
worked better back then for what seems to have been the original purpose of
Lisp, but they may also have squinted and limited themselves via convention,
not sure.)

This doesn't mean that there's no usefulness for the 'first-level parsing
step' that gives you s-expressions: in some cases you might be handling a
context insensitive language like yours. Also, it makes extending the language
much more painless (macros easily compose, or more to the point, you can
easily invent ad-hoc new languages on top of s-expressions used as inputs for
macros, and won't (ever, I think) have problems with their boundaries becoming
ambiguous).

~~~
taeric
I thought I was playing somewhat loose with the term parser. The code I used
from SICP is walking the code, after all. More accurate to just say no lexing
and tokenization?

~~~
pflanze
Hmm, true you're walking the code already. What you don't do is look at the
lexical context or maintain it while walking.

S-expressions in full Lisp don't specify lexical constructs, it is only the
pre-cursor of such (people often say that S-expressions are the AST of Lisp,
that's not true if by AST one understands to mean that each element's
functionality is clear, e.g. if it's a function call or function abstraction
or constant etc.; an S-expression only specifies a computation given a lexical
compile time context).

What you (and SICP) are doing is look for symbols like +, regardless of
whether + is bound to the addition function or something else. That is fine if
that's your intention (it's certainly easier to implement a code walker for
and you can often get away with it). If playing/deriv received

    
    
       (+ (flet ((* #'+)) (* x 2)) 12)
    

instead of

    
    
       (+ (* x 2) 12)
    

then it would need to be able to handle the flet form, and maintain the
aliasing of * to + in a lexical context that playing/product? has to inspect
to know if it's dealing with multiplication (and return false in this case).
(Caveat: I'm a Scheme, not Common Lisp user, I hope I got the syntax right
above.-- Edit: I didn't; this apparently violates CL because it tries to
shadow a standard function name, additionally the flet form apparently
requires function definition syntax not just value, i.e. (flet ((* (a b) (+ a
b)))..). So, my argument becomes a bit moot in CL as the basic operators can't
be shadowed anyway, except I'm told that you can define them in a new package
then import them, but then I'm not sure you can rebind them lexically. Scheme
is beautifully simple in comparison, it doesn't try to prevent you from
shooting your foot but of course encourages to learn to deal with the
consequences, which is why I wanted to write my post. It will still be true
for the general case in CL, too.)

Most programming languages parse from strings (possibly via tokens) to an AST.
Lisp parses from strings (possibly via tokens) to S-expressions then to an
AST. I just want to point out that one has to be careful here, and want to
point out that Lisp isn't fully parsed in the sense that one has got an AST,
when you're at the s-expression stage.

> More accurate to just say no lexing and tokenization?

The read function does more than tokenization, it doesn't give you a stream of
tokens (that includes open and close paren token), instead it does give you a
hierarchical expression representation. Yes, you don't need to tokenize, but
even don't need to do more than that. I don't know if there's some CS term for
that, in Lisp it's just "read".

~~~
taeric
That all makes sense, and I tried to talk towards this when I referred to
putting the rest of the syntax required to define a function.

That is, I have fallen to the trap of claiming lisp has little syntax before.
The idea being that I can walk the tree, so I'm walking an AST. But i realize
now that is abusing the S in AST. There is plenty of syntax there that can be
checked.

I would like to not steer others down that trap. Thanks for expanding in it
here!

~~~
kazinator
Lisp languages have lots of abstract syntax. What is lacking is the
proliferation of surface syntax.

Typically, programming languages have some dedicated phrase structure rule for
recovering each kind of AST node shape out of tokens. Moreover, AST node
shapes for which there is no phrase structure in the grammar cannot be
instantiated from source text.

Lisp languages tend to have a small surface syntax that is general, and
expresses all possible AST node shapes, including ones that have no assigned
meaning.

So, Lisps have plenty of abstract syntax; but abstract syntax is better
because it is a data structure that _usually_ no longer exhibits
representational issues that require parsing.

Sometimes it does. Lisp parameter lists like (a b &optional c &key d e)
require a bit of parsing; they are a flat list that has to be scanned for the
lambda list keywords like &optional and &key. Nothing stops Lisp programmers
from creating similar parsing problems of their own, in the arguments of a
macro. Usually it's a far cry from having to deal with a full blown grammar.

~~~
lispm
There are many cases where parsing of Lisp code is necessary. A compiler will
need to parse it. A compiler might even generate an internal AST
representation (Example: [https://github.com/clasp-
developers/clasp/blob/master/src/li...](https://github.com/clasp-
developers/clasp/blob/master/src/lisp/kernel/cleavir/ast.lisp) ). An
interpreter will need to understand the syntax, to be able to traverse code. A
tool like a code walker will need to understand Lisp syntax and even may
create an AST representation ([https://gitlab.common-lisp.net/cl-walker/cl-
walker/blob/mast...](https://gitlab.common-lisp.net/cl-walker/cl-
walker/blob/master/src/handlers.lisp)).

~~~
kazinator
Recovering an AST representation from the source code doesn't require parsing
in general, only in some specific cases. By parsing I mean "recovering of
nested syntactic units from a flat stream of tokens". E.g. when a compiler is
processing _let_ , it doesn't have to first process phrase structure rules to
obtain _let_ as a unit out of a stream of tokens, and identify its constituent
pieces like the bindings and body. It has those pieces already in a fixed tree
shape. In some constructs, the arguments of a form do have to be treated like
tokens and subject to parsing actions to identify the constituent pieces,
though usually nothing like a full blown LALR(1) grammar. _loop_ has to parse
its clauses; lambda lists have to be parsed; CL's _defun_ needs some light
parsing due to some design quirks in the syntax; _formatter_ has to parse the
format string and such. Identifying the optional declarations at the start of
a body is a piece of light parsing.

~~~
lispm
> fixed tree shape

But it does not know what it is. It has to know the structure of the LET. The
LET expression has no tagging of the parts. Is that list after the binding
list a declaration or not? How many of them? Is the symbol in the list after a
LET a variable? The s-expression does not say what a symbol is: a variable or
not. We also don't know what the relationship of the declaration to this
variable is: is there a declaration for this variable?

The relatively simple syntax of LET: let ({var | (var [init-form])} _)
declaration_ form*

We then see that the syntax for 'declaration' is quite a bit more complex.

An AST will usually represent such information.

> parsing I mean "recovering of nested syntactic units from a flat stream of
> tokens

in Lisp this is: recovering of nested syntactic units from a nested list of
tokens.

~~~
pflanze
This, together with taeric's question about tokenization made me realize that
we can think of it this way:

Macros are not receiving AST nodes [1]. They are not receiving tokens either,
though. But something in-between: how many tokens it receives is strictly
delimited (by the end paren which ends the macro form, but the macro doesn't
see an end paren token, just doesn't get more items in the list, thus can't
violate the scoping). Lisp _forces_ open and end parens to be understood as
hierarchical grouping indicators for (any kind of) syntax. This simple change
over a stream of tokens seems to happen to simplify language creation via
macros a lot.

[1] quite definitely not before macro-expanding their arguments. After macro-
expansion, by grouping the resulting s-expr with a context and a correct tree-
walker, I guess it could be said to be one.

~~~
lispm
> After macro-expansion

macros just return s-expressions - no AST data

~~~
pflanze
I know. Sorry for my messy terminology. (Not sure whether the points I'm
making here are useful or just wasting your time, I'll try to explain just to
clarify what I meant.)

I didn't mean calling the individual functions implementing macros (apparently
called "macro functions"), but something like macroexpand-all (I didn't
realize that this is apparently not portable). What I am thinking of is
something like:

    
    
        (defmacro foo (expr var &environment env)
          (let ((expr* (macroexpand-all expr env))
                (var* (macroexpand-all var env)))
            (derive-by expr* var*)))
    

where derive-by is a code walker that understands core CL, and does derivation
as part of the walk (perhaps implemented with the help of one of the code
walker libraries).

expr* (same for var* ) is not an AST in the sense of e.g. a tree of CLOS
objects where each one determines a syntactical construct even when standing
by itself; only together with env and a walker that understands the "core" CL
syntax correctly, it could, "I guess", be considered an AST--sure, maybe let's
avoid confusion and call it proto-AST instead.

    
    
          (let* ((protoAST (cons expr* env))
                 ;; could do:
                 (AST (protoAST->AST protoAST)))
             ..)
    

where protoAST->AST is a code walker that converts expr* to a tree of objects
from a CLOS hierarchy representing each kind of node unambiguously without the
need for env. protoAST and AST hold the same information content, which is why
I said "could be said to be the same".

~~~
lispm
> protoAST and AST hold the same information content

Not really. protoAST lacks the knowledge the specific code walker adds. For
example there is no way to find out whether FOO is a variable or a function
without actually 'parsing' the code. If one uses a different syntax in the
code walker, then the conversion generates a different parse.

~~~
pflanze
> protoAST lacks the knowledge the specific code walker adds

Yes, that's why I actually had said "by grouping the resulting s-expr with a
context _and a correct tree-walker_ ". True, you could make that

    
    
       (let* ((protoAST (list expr* env core-CL-walker))
              ..)
         ...)
    

to be explicit. core-CL-walker changes only when the definition of the "core-
CL" [1] changes, though, so it's probably a constant for practical purposes.

[1] I don't know whether there is any standardization of such a thing.
Probably not since macroexpand-all also isn't standard.

------
jhallenworld
For comparison, here is symbolic derivatives in C. I wrote this in 1990 for
school project:

[https://github.com/jhallen/joes-
sandbox/tree/master/x/xquiz](https://github.com/jhallen/joes-
sandbox/tree/master/x/xquiz)

I need to make slides and a better write up for it, but the program is a math
quiz with equation editing and graphing capabilities for X. It includes a
simplifier which will take derivatives if they are in the form: D(x^2)/D(x)

The simplifier also combines terms, reduces fractions, etc. I was shooting for
something like Macsyma.

The derivative taking function is here: [https://github.com/jhallen/joes-
sandbox/blob/master/x/xquiz/...](https://github.com/jhallen/joes-
sandbox/blob/master/x/xquiz/simplify.c#L208)

Equations are parsed into LISP-like lists, defined in box.h. There are list
nodes, numbers and symbols. They are allocated in aligned pages. To
distinguish between them, you look at the page header. Some of the classic
LISP implementation used this same technique- I was interested in it at the
time, but wouldn't use it today.

As you type equations in, they are typeset in standard math notation on the
screen. You can edit the tree by selecting any part of the equation with the
mouse. Once something is selected, you can type in a replacement, or click
simplify, distribute, or factor. I had fun correlating mouse coordinates with
tree locations.

~~~
taeric
Nicely done! I'm curious if you used the SICP or similar literature as
inspiration, or if this was just a natural way that you thought to do it.

~~~
jhallenworld
I don't remember SICP, but I read lots of similar books.

Actually the advisor wanted me to do this in Prolog, with the idea that we
would try to prove equivalency between the entered answer and known answer.
The problem is that I saw no way to do all of the mundane tasks in Prolog.
LISP would have been a far better choice (perhaps with a unification library),
but even LISP at the time did not have great support for X. I already knew C,
and had to get the project done...

~~~
taeric
It occurred to me later, that the first volume of Art of Computer Programming
has a derivative calculator algorithm. Even implements part in its assembly
language.

Regardless of any of that, really cool project. Being for a class underscores
the deadline you did that in. Thanks for sharing!

------
taeric
This came from a thought I had after posting
[https://news.ycombinator.com/item?id=18312529](https://news.ycombinator.com/item?id=18312529)

Any feedback is greatly welcome. Both in areas I have wrong in the
programming, and in general presentation. Thanks!

~~~
meken
Great post! A few observations.

\- It seems like the difficulty level ramps up really quickly starting with
the "So, what makes the lisp example different?" section. This was jarring
enough that I pretty much glossed over it (too much effort required to process
all the definitions at once).

As a more meta point, I wonder if symbolic differentiation is in fact a good
example of highlighting lisp's differentiating features? One could write the
same code in python/javascript with using lists and strings and recursion in a
similar way, no?

Not that I have a better example to highlight lisp's differences, which I
understand to be macros. I've read SICP and have written a fair amount of
elisp and still have not once identified a good use case for writing a DSL nor
even a macro for myself.

~~~
fiddlerwoaroof
One thing I’ve done recently in Common Lisp with macros was write a suite of
macros to make working with objective-c in Common Lisp less burdensome: with a
couple reader macros, I can almost copy-paste objective-c code samples into my
Common Lisp program and have them work.

~~~
fiddlerwoaroof
Also, some of the biggest complaints about redux and other JavaScript
libraries is the amount of boilerplate you have to write, leading to really
complicated scaffolding and code generation utilities: Babel, for example, is
essentially just a big macroexpander for JavaScript enabling, among other
things, the use of language features as soon as they are standardized rather
than having to wait years until a sufficient portion of the available browsers
support the new feature.

