
Yes, code is data, but that's not what makes Lisp cool - lisper
http://blog.rongarret.info/2018/02/yes-code-is-data-but-thats-not-what.html
======
kqr
There is one more thing that frequently comes up -- for good reason -- but is
not really a feature unique to Lisp, only a strong part of Lisp culture: live
coding, inspection, self-documentation and hotswapping, or what we call
hackability.

If you don't like how some Lisp application works, you can frequently just
replace the contentious parts as the program is running. If something does not
seem right, it is trivial to inspect all parts of the internal state. If you
don't understand a variable, you can look up its documentation inside the
running program.

It's a fantastic experience.

~~~
hippich
Could you point total newb to tutorial showing these off?

~~~
taeric
Would you be ok seeing an interactive session in emacs? If so, I can easily
get you an example created of that.

~~~
michaericalribo
I sure would

~~~
taeric
I'll work to get a basic video by tomorrow. Nothing fancy, just showing how
you can immediately define and redefine things in an emacs session. (I have
aspirations to use this as an excuse to fix something in how org-mode connects
to postgres databases...)

~~~
taeric
I grossly overestimated my chances of getting on a computer this weekend. To
the extent that I didn't. :(

So, I didn't get a video made. My humble apologies.

Rather than just leave this hanging with no response, I'll still try and get a
video while at the office this week. In the meantime,
[https://www.youtube.com/watch?v=VEcobuYr5wg](https://www.youtube.com/watch?v=VEcobuYr5wg)
shows a much more impressive example than what I was planning. This one
actually shows "live coding" with a webpage. I was just going to show how the
help system and some basic debugging in emacs works.

------
torstenvl
This is good! Why did you omit the analogy that code-manipulation-in-Lisp :
code-manipualtion-in-C :: webpage-manipulation-via-the-DOM : webpage-
manipulation-via-regex ?

I thought it was very effective.

~~~
lisper
Oversight. I wrote this in a hurry. Maybe you can post that as a comment on
the blog.

------
lispm
I don't think programs are usually strings.

Most programs I see are text files. If I edit them in an editor, I don't know
what data structure represents them, but it might be a doubly linked list of
strings, or some other data structure.

If a compiler reads a program, it uses typically a stream of characters.

A string would be something like this: a delimiter, some characters, the
delimiter. Example: "a + b". It could also be a string data object in some
program.

But programs are not always represented as strings and many are not. Sometimes
they are. For example an EVAL procedure may take a string as an argument -
then code is represented as a string or passed at runtime as a string object.

When we say in Lisp code is data, we actually mean code is data, but not in
the trivial form of text representation formats (text files, strings, ...).

It's an abbreviation. We expect usually not to mention, that it is not of the
trivial version. Like when we speak of a picture with lots of color, we don't
mention that this excludes pictures with only black, white and shades of grey.

Generally, if I see

    
    
      a + b
    

then it says nothing about a data structure and its syntax to represent that.
All I know that it must be a syntactical legal expression in some programming
language.

If I see Lisp code in the form of

    
    
      (+ a b)
    

I know that this uses the data syntax for s-expressions to represent that
code.

I then would require that this is syntactical legal Lisp - THIS requires that
in an s-expression, the + operator comes first - that it uses prefix syntax.

Wien we mean code as data, we mean that Lisp uaes an actual and explicit data
format to encode its programs: s-expressions. The data format is similar to
JSON or XML, in that it has an explicit external representation. It's also
similar to JSON, because both have an internal representation in a programming
language.

If we would use JSON as a format to encode programs, we would also be able to
say, code is data.

S-expressions are slightly more effective as data encoding for programs than
JSON, because it uses less delimiters and encodes a rich set of data
structures.

So, yes. Code with a non-trivial and simple data-representation makes Lisp
cool.

~~~
xfer
> Wien we mean code as data, we mean that Lisp uaes an actual and explicit
> data format to encode its programs: s-expressions.

Every programming language uses an "actual" and explicit data format to encode
its programs, the valid programs all parse to the corresponding AST.

The only thing unique about Lisp here is a construct to parse the
serialization format(s-expressions) and perform actions on it i.e. macros. The
fact that parsing s-expressions is so trivial makes it seem like manipulating
code as data is somehow unique to Lisp.

~~~
lispm
> Every programming language uses an "actual" and explicit data format to
> encode its programs

where? Show me the data format which is used by C programmers to encode
programs.

A data format with an external and internal representation, and both used by
the programmer.

An AST of a C compiler isn't it, since the programmer isn't using it. Neither
externally as text, nor internally as data. It's only used internally by a
compiler - if it does use one. The C programmer does not write code with ASTs,
nor does he even manipulate code via ASTs operations.

It's not that some languages have ASTs -> they are simply no neutral data
format and the programmer isn't using this to write/read programs. In some
languages or implementations ASTs are available to the programmer, but this is
after a tool (the compiler) has parsed the program according to the language
syntax rules. But that is not what Lisp does, s-expressions are more like a
general data format which by the Lisp reader (a tool) is used to translate
into a token tree. But the Lisp reader is not a compiler internal stage. It's
a general tool, which turns externalized data into internalized data.

ASTs are also a different thing from s-expressions, since s-expressions don't
represent abstract syntax of Lisp programs. S-expressions are language
agnostic: it does know nothing about the syntax of Lisp constructs and doesn't
represent no syntactic information. On the s-expression level (a b c) is no
different from (+ 1 2) or (defun foo ()).

The C syntax isn't layered on top of a neutral data format, like Lisp is.

~~~
xfer
> In some languages or implementations ASTs are available to the programmer,
> but this is after a tool (the compiler) has parsed the program according to
> the language syntax rules. But that is not what Lisp does, s-expressions are
> more like a general data format which by the Lisp reader (a tool) is used to
> translate into a token tree. But the Lisp reader is not a compiler internal
> stage. It's a general tool, which turns externalized data into internalized
> data.

So, what additional power Lisp has as "code as data" vs the language that
gives power to manipulate ASTs easily(with quasi-quotation)?

>ASTs are also a different thing from s-expressions, since s-expressions don't
represent abstract syntax of Lisp programs. S-expressions are language
agnostic: it does know nothing about the syntax of Lisp constructs and doesn't
represent no syntactic information.

Irrelevant to the discussion of "code as data". Every language can read s-expr
in language agnostic way.

> The C syntax isn't layered on top of a neutral data format, like Lisp is.

I never said that the languages are built on top of a neutral data format, but
asking the benefits of being so. Unless i am mistaken, the only benefit is
being easy to parse by tools(editors, compilers etc) and with quasi-quotation
the power of macros.

~~~
lispm
> power to manipulate ASTs easily(with quasi-quotation)?

From a point of Lisp code manipulation, quasi-quotation does not reach far and
thus the convenience it provides is only shallow. It's useful to have, but its
main purpose is to build up code based on some template mechanism. There are
other things which are also important, like destructuring of code, which a
typical macro mechanism will support in some way.

Another point: to have an AST you need to have the code parsed already. The
AST is an output of a parser stage. That means you are bound to the syntax
this parser implements and this AST can represent - or you need to be able to
feed this parser with different syntax definitions and have it runtime
extensible. Since Lisp macros see only code as internal data, the input format
is much less limited. It can also transform arbitrary data into code. Or you
need to be able to deal in some other with code which does not conform to what
the parser expects.

> Irrelevant to the discussion of "code as data". Every language can read
> s-expr in language agnostic way.

'can' vs. usually done is a huge practical difference. s-expressions are not a
standard input format for C compiler. I have not seen s-expressions or a
similar data format in the C language standard used to represent code. I'd
would be interested to see a reference to it, if you have one.

> I never said that the languages are built on top of a neutral data format,

But Lisp is.

> Unless i am mistaken, the only benefit is being easy to parse by
> tools(editors, compilers etc) and with quasi-quotation the power of macros.

Quasi-quotation is only a convenience feature - which is great, but they don't
add any power to the language. Lisp can create the same effects without quasi-
quotation.

The benefits are:

* operations on the data format can be supported in an text editor and then can be applied to both data and code

* the code structure is not limited to what syntax a programming language parser implements or what an AST can represent

* the manipulation of internal code as data is convenient and simple for many cases, since the typical operations can be applied

* the internal format used by Lisp has a default external format, but can also be used with different external formats

* the programming language implementation exposes this internal data in various stages to the user, which it then can conveniently use it. For example macros see this data and can transform it with the usual tools. Interpreters use it and can expose it to the developer. The compiler can use it as input. The pretty printer takes it as input to create textual representations.

The main point is that it is convenient to use. Thus code transformations and
code manipulations of s-expressions are both common and simple in Lisp.

The effect is that user-written code manipulation is widely used in Lisp -
even at runtime, where for example in C it is typically done in external tools
in preprocessor stages or hidden from the user at compile time.

~~~
xfer
> There are other things which are also important, like destructuring of code,
> which a typical macro mechanism will support in some way.

Destructuring doesn't require it to be a list/s-expr. You can destruct ASTs in
any language that supports pattern-matching.

> or you need to be able to feed this parser with different syntax definitions
> and have it runtime extensible.

That's what languages should aim to do. For example rust's procedural macros
can transform a TokenStream to another TokenStream and doesn't have to satisfy
Rust language parser. Although that doesn't extend to lexing.

> 'can' vs. usually done is a huge practical difference. s-expressions are not
> a standard input format for C compiler

I am not talking about a specific language compiler, but modern languages are
heading in the right direction without using s-expr.

> Quasi-quotation is only a convenience feature - which is great, but they
> don't add any power to the language. Lisp can create the same effects
> without quasi-quotation.

Yes, but then you write your own macro-expander.

> operations on the data format can be supported in an text editor and then
> can be applied to both data and code

Yes, but they will not be very helpful without additional tooling.

> the code structure is not limited to what syntax a programming language
> parser implements or what an AST can represent

And you have lost the previous point, now your "code as data" is not so well-
supported.

> the programming language implementation exposes this internal data in
> various stages to the user, which it then can conveniently use it. For
> example macros see this data and can transform it with the usual tools.
> Interpreters use it and can expose it to the developer. The compiler can use
> it as input. The pretty printer takes it as input to create textual
> representations.

True, but these all can be provided without s-expressions, the compiler just
has to be designed properly into reusable modules.

Rest of your points just suggest that the tooling part is easier and simpler
to do and understand, which ofcourse i agree with. But not all programmers
prefer Lisp like syntax and most languages haven't really cared about it or
see as undesired(macros bring some undesirable properties as languages like
loop pop-up) flexibility.

~~~
lispm
> Destructuring doesn't require it to be a list/s-expr. You can destruct ASTs
> in any language that supports pattern-matching.

It's not about requiring. It's provided in Lisp. Macros already provide
destructuring in their arglists.

> That's what languages should aim to do. For example rust's procedural macros
> can transform a TokenStream to another TokenStream and doesn't have to
> satisfy Rust language parser. Although that doesn't extend to lexing.

A tokenizer is not a language parser.

> I am not talking about a specific language compiler, but modern languages
> are heading in the right direction without using s-expr.

That's nothing new or modern. Code transformations and code representation has
been used for decades.

> Yes, but then you write your own macro-expander.

No, quasi-quotations have nothing to do with a macro-expander. It's a general
feature in Lisp - independent of macros. One alternative is to procedurally
build up code.

There is no functionality difference between

    
    
      `(a ,a)
    

and

    
    
      (list 'a a)
    

Both compute the same list structure.

>> operations on the data format can be supported in an text editor and then
can be applied to both data and code

> Yes, but they will not be very helpful without additional tooling.

There is nothing wrong with tool building.

>> the code structure is not limited to what syntax a programming language
parser implements or what an AST can represent

> And you have lost the previous point, now your "code as data" is not so
> well-supported.

But I gain a lot of flexibility.

> True, but these all can be provided without s-expressions, the compiler just
> has to be designed properly into reusable modules.

That's what Lisp already provides.

> Rest of your points just suggest that the tooling part is easier and simpler
> to do and understand, which ofcourse i agree with. But not all programmers
> prefer Lisp like syntax and most languages haven't really cared about it or
> see as undesired(macros bring some undesirable properties as languages like
> loop pop-up) flexibility.

All true.

~~~
xfer
> A tokenizer is not a language parser.

It's not a tokenizer it can transform a token stream to another token stream
with a given function which can be some language parser/just rust code
producing desired rust ast.

> No, quasi-quotations have nothing to do with a macro-expander.

My bad, i was confusing it with mixing evaluation.

------
ljm
For years I've tried to play with Lisp languages: reading books like Beautiful
Racket, experimenting with Clojure/ClojureScript, etc. and it's often
conflicted with what I'm trying out in other languages to understand different
approaches to programming (Rust, F#, Haskell...).

Almost every year I come crawling back to emacs and I start putting the pieces
back together (first, by hand, then Spacemacs a couple of times, then by hand
again but in a different way). It has such an incredible allure and it's a joy
to write. I don't know what else I would write in Lisp right now so I just
abandon my config and rebuild it to get that feeling again.

------
taeric
Fun read. Though, I suspect this is what it feels like to be in a choir. :D

Small critique on presentation. The title clearly indicates there are two
parts to this, a brief exploration on "code is data", followed by a dip into
what makes lisp cool. Having some visual indicator saying "we are now at the
cool part" would be good.

------
AnimalMuppet
That was a great explanation. But if I understand this correctly, you're
actually agreeing that homoiconicity is (at least a big part of) what makes
Lisp cool, but you're avoiding using the word.

Or, perhaps, you're saying that it's not _just_ homoiconicity. It's that it's
the _right_ homoiconicity - lists. Lists are well-suited for being processed
as data in Lisp (kind of the point of Lisp, really), and lists are great for
representing code (abstract syntax trees), _and_ lists are really simple to
parse if they're represented cleanly.

~~~
lisper
> it's not just homoiconicity. It's that it's the right homoiconicity

That is exactly right.

------
nocman
> First, the idea that "code is data" is a popular aphorism, but even the most
> casual reflection on what this slogan actually means will reveal that it
> can't possibly be the right answer.

I disagree with this statement. Support for your argument seems to stem from
the other statement you then make: "What makes Lisp cool is not that programs
are data (because all programs are data), but that they are a particular kind
of data.".

Personally I think it's not a good argument, for the following reason: I would
guess that 99% of the people who argue that "code is data" is what makes Lisp
cool, mean exactly that -- that code is expressed as lists of lists (and they
are thinking of all the nice implications of that).

So effectively, you are saying that the statement is invalid because it
_could_ mean something else, but almost all of the people who make that
statement are making the exact same argument you are for why Lisp is cool.

I opened your article expecting to find an explanation of why some other
feature or set of features of Lisp was _way_ more important/useful than just
code being written as lists of lists, only to find that you were arguing for
the importance of that exact feature.

So while it is worth pointing out that having code as a list of lists has
awesome implications, I guess I would have gone about discussing in a
different way.

~~~
AnimalMuppet
I don't think he's trying to persuade people who already buy Lisp. And for
everyone else, they can get hung up on "code is data" if it's presented as a
"Lisp only" kind of thing, so he avoids that whole discussion and comes at it
another way.

~~~
nocman
As I said in my reply above, I'm all for the clarification of what Lisp
programmers mean by "code is data". I just disagree that the fact that someone
may misunderstand what Lisp programmers mean when they use the phrase somehow
makes it an invalid thing for them to say.

"code is data" has a specific meaning to Lisp programmers, and it is
completely valid.

------
bsder
> Parsing S-expressions is super easy.

No, it's !@#$%^& not.

This pisses me off every single time I see it.

Parsing s-expressions requires on-the-fly allocation of memory and something
resembling recursion.

If you can't allocate memory--good luck.

If you don't have recursion in your language, you're going to have to build it
yourself--good luck.

And god help you if you have dot notation--(a b) is a _tree_ of (a . (b .
null)) while (a . b) is an actual pair.

Clojure shooting the idea of building things out of cons pairs and using lists
composed of arrays is one of its most important design choices.

~~~
phoe-krk
_Parsing s-expressions requires on-the-fly allocation of memory and something
resembling recursion._

Parsing _any_ code requires on-the-fly allocation of memory because your
function can contain 10 or 100 or 1000 lines of code and, when you are at the
beginning, you don't know how long the whole function is.

Parsing _any_ code requires something resembling recursion because your
function can contain 3 or 10 or 20 nested scopes and, when you are at the
beginning, you don't know how deep the whole function is.

Of course that you can say that you can pre-read the code and compute how much
memory you need beforehand, but that applies to Lisp as much as it applies to
any other language.

~~~
bsder
> Parsing any code requires on-the-fly allocation of memory because your
> function can contain 10 or 100 or 1000 lines of code and, when you are at
> the beginning, you don't know how long the whole function is.

This is not necessarily true. Parsing a language can be bounded by the
grammar. If you don't have indefinitely nested structures (like arithmetic
open parentheses), your parsing is linear and you can bound the memory (ie.
your parsing memory cannot be longer than your input string, for example).

I believe that Tcl and Microsoft Basic fell into this category. (I believe
that Ousterhout used to actually generate code in the input buffer behind the
current parsing point precisely because Tcl was _designed_ to do this).

> Parsing any code requires something resembling recursion because your
> function can contain 3 or 10 or 20 nested scopes and, when you are at the
> beginning, you don't know how deep the whole function is.

Indefinitely nested lexical scoping is not a requirement for a language.
Famously _Javascript_ screws up all manner of programmers because they expect
that to be true, but it is not. And many languages start out with a single
global scope and grow lexical scoping only after they gain some complexity.

~~~
pubby
I'm a little confused at where you're coming from. Do you program
microcontrollers or something? Since when has recursion or the ability to
allocate memory been an issue? 30 years ago, maybe?

I understand lisp isn't trivial to implement if all you have is assembly with
no OS or libraries, but beyond that it's pretty damn easy. Even in C it only
takes a few dozen lines.

> Javascript

I'm not a JS programmer, but I think JS has had infinite lexical scope at
function-level since the beginning. It does not have scope at block-level,
which is the confusing part.

------
erikj
I think some charts would communicate the point better than pure text in this
case.

------
cgearhart
This was very helpful. Thanks.

~~~
lisper
My pleasure. Thanks for the kind words.

------
mattgreenrocks
This was great, thank you!

------
carapace
The name LISP derives from "LISt Processor". ;-P

Very nice article.

~~~
anon04532
It's actually Language of Irritating Superfluous Parenthesis.

Anyway, even saying "code is linked list" does not really cut it as you can
represent 'ordinary' programming language program as a list of tokens. What
makes LISP special is that code is a first class citizen and the mapping
between code you edit and the programs run-time memory representation is
trivial. Combined, you get this special feeling that "code is data".

~~~
carapace
Well, I think lisper nailed it in the article: It's not _just_ that the code
is represented as data structured in linked-lists, it's that that data exposes
the _semantic structure_ of the code in ways that are more pragmatically
useful (among other nice properties) than char strings.

When you get into the LISP interpreter itself, which is just a linked-list
structure describing how to process linked-list structures (like itself) as
code, well... Here's William Byrd on "The Most Beautiful Program Ever Written"
(ye LISP interpreter)
[https://www.youtube.com/watch?v=OyfBQmvr2Hc](https://www.youtube.com/watch?v=OyfBQmvr2Hc)

