

An editor for composed programs - ltratt
http://tratt.net/laurie/blog/entries/an_editor_for_composed_programs

======
tikhonj
Emacs's structured editing, in the style of paredit[1], sounds very similar to
a text-based version of syntax-directed editing (SDE). Designed for Lisp and
S-expressions, it's a system that lets you edit and navigate around nested
expressions at a syntactic rather than textual language. As you type, the
correct level of nesting is maintained and there are commands for moving and
editing code in terms of s-expressions rather than characters.

In practice, it's very easy to start using and very powerful... but requires
using Lisp. It'll seem odd if you're not regularly a Lisp user.

Happily, this idea can generalize well enough to distinctly non-Lisp
languages. Structured Haskell mode[2] does this for Haskell, which has a less
regular and sparser syntax than s-expressions. The documentation has a bunch
of handy animations showing the various features, giving you an idea of how it
works without having to install it yourself.

This doesn't address the _meat_ of the blog post—composing languages and
grammars by taking advantage of structured editing—but it shows that the
necessary foundation is _possible_ with a perfectly normal text editor. It
would be interesting to see if some of the ideas from Eco could be added to
structured editing modes like this without requiring you to use an external,
self-contained editor.

[1]:
[http://www.emacswiki.org/emacs/ParEdit](http://www.emacswiki.org/emacs/ParEdit)

[2]: [https://github.com/chrisdone/structured-haskell-
mode](https://github.com/chrisdone/structured-haskell-mode)

Also, here's a great video of Paredit in action:

[https://www.youtube.com/watch?v=D6h5dFyyUX0](https://www.youtube.com/watch?v=D6h5dFyyUX0)

~~~
kazinator
... and of course, in Lisp you can embed other languages in terms of their
syntax trees, so the composed language problem goes away. Your HTML just looks
like (html (head ...) (body (h1 ...))) or whatever; no "language box for HTML"
needed.

~~~
mtdewcmu
>> Your HTML just looks like (html (head ...) (body (h1 ...))) or whatever; no
"language box for HTML" needed.

In your example, the language composition is smooth because there's only one
language there: Lisp. HTML has capitulated.

~~~
willismichael
That's the point, HTML is just a hobbled lisp with an awkward syntax. If
you're building a template engine in a lisp, it's much nicer to just use the
native lisp syntax instead.

~~~
judk
HTML is a subset of lisp with a different syntax. The rest of your point holds
without lashing out at HTML.

In different environments, different syntaxes show strength. What happens if
you misplace a ) vs misplacing an </tag> ?

~~~
kazinator
What happens when you misplace a tag, and it's around 1995, is that browsers
somehow parse it anyway without complaining, rendering the page anyway. Users
then believe that this is the correct syntax: closing tags is optional, as is
the order in which it is done. Twenty years later, you end up with a mess of a
situation where every browser has to support not only its own historic buggy
behaviors but those of other browsers.

You cannot "misplace" a closing parenthesis, at best you can have too many of
them not enough. Of course, you can enclose the wrong amount of material, but
to see what is being enclosed, you just need parenthesis matching support in
the editor. This is more common than XML tag matching support. E.g.

[http://stackoverflow.com/questions/500989/jump-to-
matching-x...](http://stackoverflow.com/questions/500989/jump-to-matching-xml-
tags-in-vim)

------
al2o3cr
Interesting idea, but the code example used completely puts me off - it's
literally the _opposite_ of well-factored code, with HTML, Python and SQL all
smeared together.

------
tephra
Here are the slides [1] from ECOOP 2014 presenting his research. It was a
really interesting talk. He also did a demo which was cool.

[1]
[http://ecoop14.it.uu.se/programme/ecoop_ss_tratt.pdf](http://ecoop14.it.uu.se/programme/ecoop_ss_tratt.pdf)

~~~
sparkie
Additionally, they published this paper last year with an early demo of the
editor linked on Youtube. I've posted it on here a few times but it hasn't had
much response.

[http://soft-dev.org/pubs/pdf/diekmann_tratt__parsing_compose...](http://soft-
dev.org/pubs/pdf/diekmann_tratt__parsing_composed_grammars_with_language_boxes.pdf)

[https://www.youtube.com/watch?v=LMzrTb22Ot8](https://www.youtube.com/watch?v=LMzrTb22Ot8)

------
Guvante
Interesting idea, and any research into getting away from text editors is
positive.

However in order to compete with text, which has a bountiful number of editors
available. You need to beat the programmers favorite editor at writing the
program. Doing this given the competition size is incredibly hard.

While I think the has merit at a high level, poor implementation isn't the
biggest thing holding back non-text implementations. Being the best editor for
everyone has been.

On a related note, is there a reason they didn't go with some form of
escaping? Similar to CDATA in XML.

~~~
mtdewcmu
>> any research into getting away from text editors is positive.

Any research at all? I'm curious: what it is that's unworkable about text in
your view?

~~~
Guvante
> Any research at all?

Other people spending money to figure out if an alternative idea will work is
good for me.

> what it is that's unworkable about text in your view?

Nothing, it just feels like a local maximum. Typing is incredibly efficient,
so I don't think it is going anywhere.

There are plenty of ways of showing that in a certain situation non-text can
be more efficient, so I believe there is a way to introduce non-text that can
make the general case more efficient.

------
jimbobimbo
Here's how Visual Studio 2013 handles HTML, C# and JavaScript syntax coloring
in one Razor view:
[http://i.imgur.com/he1ieG9.png](http://i.imgur.com/he1ieG9.png)

------
npatrick04
I've been using Org Mode with emacs for writing literate programs. You could
write a simple front to Org Mode to decrease the clutter...but the
hierarchical nature of the Org Mode files tends to reduce the issue of clutter
already, so it may not be worthwhile.

[http://orgmode.org/manual/Working-With-Source-
Code.html#Work...](http://orgmode.org/manual/Working-With-Source-
Code.html#Working-With-Source-Code)

------
kazinator
> _" In a sense, when programming in a text editor I think in trees; mentally
> flatten that tree structure into sequences of ASCII characters that I can
> type into the text editor; and then have the parser recreate the tree
> structure I had in my head when I wrote the program."_

This is quite an amazing article in that the author really "gets" Lisp and yet
at the same time doesn't get it.

(It isn't mentioned; don't waste time looking.)

If you admit you're already thinking in terms of that tree structure already,
just freakin' use it instead of this charade of encoding into in some
completely rearranged way that has to be unraveled back to what you were
originally thinking of anyway (or something else).

~~~
jamii
The same thing is true for lisp. Writing a tree-based interaction mode like
paredit that doesn't die horribly on malformed text is a non-trivial problem
([http://mumble.net/~campbell/emacs/paredit.el](http://mumble.net/~campbell/emacs/paredit.el)
\- 2625 lines of emacs lisp just to edit a tree).

~~~
mtdewcmu
The trend as you move away from simple sequences, such as binary numbers, to
state machines, toward trees and general graphs, is that problems become
rapidly less tractable. The situation degrades strikingly fast: numbers have
sophisticated algebras, regular languages are fairly comprehensible; but on
the other end, graphs are replete with NP-complete (intractable) problems.
Intractable problems aren't something you have to solve, obviously, because
you won't get anywhere. But I think that not only are the hard problems in
graphs intractable, the hardness of graphs carries over to the simple
problems. Nothing is simple on graphs, actually, and trees are not far from
graphs. So if unstructured text seems almost too simple, we should feel glad,
because trees are already nettlesome even to write straightforward code for. I
feel that hierarchies, as seductive as they seem, almost invariably bite; I
think it's a good principle to rule out all flat solutions before considering
hierarchies as solutions to any problem.

~~~
innguest
I feel the same way.

I had to write a library to transform JSON from one shape into another,
carrying over the values but placing them in different nodes on a new tree. I
found that thinking in terms of tree transformations was incredibly hard. The
way I solved it was by flattening the JSON, transforming the flattened
version, and when I'm done, inflating it back up.

In other words, working with the simpler, "2D" version of a JSON was much
simpler than its "3D" tree structure, and writing flatten/unflatten
independent from the transforming code was a nice modularity bonus.

It's really all about decomposing the problem and solving its parts.

~~~
kazinator
If you think in terms of the operations that perform the tree transformations,
of course you will have difficulty because, you're using a different tree: the
syntax tree representing the code of those transformations! Not the syntax
tree of the source and destination structure. So you are "one removed" from
the real problem. Moreover, since both domains are trees, it's confusing.

It sounds like you could have benefited from a way to express pattern matching
using JSON syntax, for destructuring a JSON object, and to represent the
synthesis of new JSON using a "JSON quasiquote".

E.g. Lisp transformation with classic destructuring-bind and backquote:

    
    
        (destructuring-bind (a (b (c &rest d) e) f (g h)) some-obj
          `((,a ,b) ,c (,@d 1 2 3 ,e) (,f ,g, h)))
    

There are pattern matching libraries nowadays that do a lot more than
destructuring-bind, but it illustrates the basic point.

The destructuring-bind macro writes the code to pull apart some-obj according
to the tree picture with embedded variables. The backquote syntax generates
the code whose evaluation synthesizes the new tree object according to a
template, with the values of expressions indicated by , and ,@ substituted and
spliced into the template.

~~~
innguest
I never thought of using destructuring-bind together with quasiquoting in such
a direct way. It does seem that that's a simpler way than what I have done.
I'll have to think about it some more. Though we have to consider I had to
write this in Ruby, where I don't have access to CL niceties. :)

It did get me thinking, though, so I appreciate the advice. I'm wondering now
how to port that idea to Ruby.

~~~
kazinator
More commonly, this destructuring is done in macros, by their "macro lambda
lists". destructuring-bind is just a binding gadget which performs almost the
same destructuring as macro lambda lists.

(Why almost: because "destructuring lambda lists" don't support the
environment parameter of macro lambda lists!)

------
pfraze
I will almost certainly be giving this a try tonight.

------
rch
nice.

