
The GNU extension language - fogus
http://wingolog.org/archives/2011/08/30/the-gnu-extension-language
======
gruseom
When it comes to extensibility, Emacs and Emacs Lisp have one of the most
compelling stories of all time. (In fact, is there anything that beats them?)
If Andy is right that Guile can do for GNU what Elisp did for Emacs, only
better, it would be downright amazing.

I wonder, though, whether some things that made Emacs/Elisp so successful
would get lost in the switchover. For example, while dynamic scoping is bad
for writing robust complex software, it does make Emacs easier to extend.

Emacs is not really an application or a platform, it is a framework: a
framework for a text editor, one that comes with a well-thought-through model
and a flagship application (Emacs itself) that doubles as a development
environment. I don't mean "model" as in MVC, but in the sense of a consistent
set of concepts that can be composed to do things. In Emacs those concepts are
things like buffers and windows and text properties. Emacs has one of the best
models I've ever seen; it should be taught as an example of how to do this
kind of design. Emacs on the surface may be clunky and unlovable, but dive
underneath and its world is beautiful.

When people extend Emacs, they're building programs out of that model using
tools designed for building programs out of the model (which themselves were
built out of the model). If you shift from Emacs to GNU as a whole, you're
moving from that single -- self-consistent and composable, not to mention
interactive -- design context into a hugely diverse environment with no common
design context at all. If Emacs is a planet, GNU is a universe. It's not clear
how well Emacs' extensibility magic can be reproduced at that level. If the
benefits come mostly from the language, Andy may be right. I hope he is. But
if the benefits come mostly from Emacs' framework and interactive environment
(both of which are inseparably bound up with the language), then the magic is
likely to be left behind with them.

Still, it's hard to argue against his case for choosing the most powerful
possible extension language so as to maximize the extending that people will
be able to do with it. To argue against this in favor of familiarity is
particularly to miss the point.

~~~
Peaker
How does dynamic scoping make Emacs easier to extend?

~~~
gruseom
_How does dynamic scoping make Emacs easier to extend?_

It gives you easy access to everything that's going on at runtime. For
example, many Emacs buffers are read-only or have read-only bits of text (e.g.
the prompt in a Slime REPL). If your code tries to change that text you get an
error. But there's a trivial idiom for getting around it:

    
    
      (let ((inhibit-read-only t))
        (do-the-thing-that-changes-text))
    

Any code invoked during the execution of this form, such as the system code
that enforces read-onlyness, will now behave differently.

Because Emacs is dynamically scoped, everything works this way. You don't
depend on the author of some module having said in the past, "Someone might
want to modify this variable at runtime someday, so I'll declare it to be
dynamic." Everything is readable and writeable by default. This is a big deal
for Emacs' extensibility.

Dynamic scoping isn't good in general, but Emacs has benefited handsomely by
making this tradeoff in its own case. There are thousands of useful Elisp
modules that interoperate with remarkably little fuss, all extending the same
system. Yet the language is dynamically scoped and has zero encapsulation.
Most people, I think, would deny that that was possible if it didn't exist.
Many people still deny it is possible even though it does!

~~~
Peaker
I don't think dynamic scoping makes it possible -- I think it is used simply
because it is there. If it wasn't there, some other solution, just as good (if
not better) would be used.

When writing e-lisp, I find dynamic scoping limiting, and using lexical-let,
especially when I want curried/partial-application, is extremely annoying.

For me, extending emacs is harder because of dynamic scoping. It is entirely
possible to have code be executed in some sort of "Reader" monad/env, and
execute it in a different than default env -- which gives you "dynamic
scoping" on whatever input you want, without contaminating every other name.

I think almost every negative feature you could possibly implement would find
"good use" that defended its existence (Dynamic scoping, $_ in Perl,
implementation inheritance, etc).

------
mindslight
Although treasonous to say, I don't think a sexp-based language is appropriate
for a wide community of programmers. (I'm currently enjoying coming back to
PLT Racket from Clojure, so please don't write me off as someone who can't get
over the initial shock.)

Macros make it _too_ easy to define new syntax, to the point where things that
should be core language features are neglected, only to be continually re-
half-implemented, etc. Take a look at what passes for 'good enough' for
destructuring in Clojure, both the older let-based and the newer core.match
(which still lacks the extensibility of Racket's match). The deficiencies can
always be fixed 'later', right?

No syntactic difference between function application and special forms means
that one requires an editor that can somehow differentiate (slime/drracket
approaches) to even have a _shot_ at quickly scanning unfamiliar code (say,
quickly looking at what an implementation actually does). It's pointed out
that Guile will handle other languages (imho this is tacitly admitting
uncertainty about Scheme). But multiple languages only stand to make the code-
reading problem even worse! Never mind that Scheme will be the only first-
class language (other languages can't consume macros) so alternatives are at
best advertisements to get people hooked and having to learn the 'real
language'.

The little bit of Ruby I've written opened my eyes to how much you can do with
just first-class code blocks, and made me understand why something like SLIB
would opt for explicit lambdas everywhere instead of macros. Macros are so
enticing to cut down on boilerplate, but they're visually indistinguishable,
and without plenty of forethought they're not composable.

Clearly many skin syntaxes for Lisps have been attempted and never caught on.
I don't currently have much to add on that front besides a suggestion of not
being afraid of well thought out concrete syntax (how many base special forms
are there, really?). But really, I think my whole post shows the main problem
with Lisp, at least for me. It's so easy to do things and it's such a blank
slate that no matter what I sit down to do, I always end up trying to create a
new language.

~~~
jrockway
_Although treasonous to say, I don't think a sexp-based language will ever be
adopted by a wide community of programmers._

The problem with this argument is that a sexp-based configuration language is
already adopted by a community of programmers: Emacs Lisp. Emacs itself ships
with over a million lines of Lisp code, and there are thousands of extensions
available from around the Internet.

The problems you worry about with respect to macros have simply not happened;
most people name macros the same way (starting with _def_ or _define-_ ), and
most macros are written "cleanly" in such a way as to not surprise the
programmer. (I'll be honest: I can't think of one time I've ever hit a but in
a Lisp program caused by a strange macro.)

Emacs Lisp shows that if you give someone a turing-complete programming
language on top of a powerful piece of software with a clean and well-
documented API, programmers will ignore the warts and enhance the software in
ways you never thought possible.

Guile is not a perfect programming langauge, and I would never use it by
choice. But if I started using some piece of software that used it for
configuration, I know I would love it because it would stay out of the way and
_let me make things_. Once you are making cool things, you don't care about
the syntax or programming language features at your disposal: you just make it
work.

Some other examples of this include JavaScript for Firefox extensions and
Vimscript for Vpim extensions. JavaScript is very ... minimal compared to
other languages like Perl, Python, Ruby, Haskell, and so on, but people write
great browser extensions anyway. Vimscript is the weirdest thing I've ever
seen but Vim is able to do a lot of cool stuff despite this.

The only thing a successful extension language needs is something worth
extending. If you use Guile as your extension language, you can concentrate
doing that instead of writing your own programming langauge toolchain.

~~~
gruseom
_I can't think of one time I've ever hit a bu[g] in a Lisp program caused by a
strange macro._

What about macros that have the same name? In Emacs, the convention is to name
your functions foo-this and foo-that if your program is foo.el. But this does
not seem to be true for macros. Is that because macros don't leap outside the
compilation context of foo.el? (Edit: nope, I checked it and they do.) Or is
it that people are sloppier about naming their macros?

Edit: or is it rather that, if your foo.el defines a macro 'blah, and I write
bar.el that says (require 'foo) and then defines its own macro 'blah, it's my
macro that gets used to compile my code, so the one in foo.el can't do any
harm to it?

~~~
jrockway
The naming convention for macros is a little weird. My rule is: if the macro
is user-visible (and not something to make your module easier to implement),
it should start with define- or with-, and should include a uniquifier
somewhere in the name. If you were writing an extension called foo, you might
provide a macro called with-current-foo or define-foo-bar.

If someone already has a macro with this name, then don't call it that.

The reason for ignoring the normal package-function naming convention is that
it's simply ugly to write code like:

    
    
       (eproject-with-current-project "foo" (whatever))
    

when you could write

    
    
       (with-current-project "foo" (whatever))

~~~
gruseom
I get (and practice) the part about user-visible macros. But what about macros
that _are_ just to make my module easier to implement? I often write those to
concisify (well, you did say "uniquifier") my code. There's a tension here:
encoding the package name in the macro name can easily make the name so long
that the whole thing is why-bother, but using a short name runs the risk of a
collision. For example, paredit.el defines XCOND. Although I don't have an
XCOND, that's just the sort of macro name I might have chosen.

------
dkersten
This quote from the comments summarizes my thoughts on this:

" _Robert says:

30 August 2011 11:55 PM

It isn't whether Guile is good, bad, usable, or not usable. It is whether
people will and do USE it. I have not seen that as the case for Guile or
Scheme in general._"

I don't know anybody anywhere who has even tried to use Guile. I don't even
know if anybody I know have ever looked at it. I don't know how many people
even know it exists.

I personally love Scheme as a language, but I honestly can't see myself ever
investing enough time into Guile to be able to write program extensions in it.
I don't see my friends (some of whom I believe would write extensions to
popular open source software if they could do so in a language that is
sufficiently similar to one they already know - I certainly would be a lot
more likely to do so and I'm not even Lisp averse) doing so either.

" _First of all, what does it mean to have a GNU system based on a non-GNU
technology? What would be left of GNU?_ "

So pick a non-GNU language and create a GNU implementation.

Personally, I think the best choice would be Lua. Its mature, a lot of people
already use it as an extension language and its very similar to a lot of
popular languages, so resistance for people to learn it is low (it is
reasonably similar to Javascript and to a lesser extent, Ruby and Python;
also, as has been proven by the game industry, it is easy to learn for C, C++
and Java programmers. Finally, while not GNU, it has some exceptionally good
implementations).

~~~
tikhonj
Well, I've never used Guile, but I _have_ used Elisp, to great success. It
allowed me--with little previous Elisp experience--to easily write a minor and
a major mode without having to spend too much time in the manual. Given that
Guile is very similar--just more modern and polished--to Elisp, I definitely
see it as a reasonable extension language.

I think the issue with the limited number of extensions written in Guile is
more with the projects that can be extended than with the language--I'm not
aware of any that I use, and certainly none that I would like to extend.

~~~
dkersten
I get the impression that the majority of emacs users are not averse to
learning seemingly weird and dificult new things (otherwise you _probably_
would not have learned to use emacs in the first place). The majority of users
are, in my experience anyway, put off by all sorts of things (and often quite
unrelated to the merits (or lack of) the language itself). I have also
experienced quite a bit of lisp hostility. All of these things add up for me
to feel there are better choices as an extension language if you want the
_average_ [1] user to write extensions.

[1] by average I mean that they have the technical skill to do so, but aren't
terribly invested in learning lots of new things. I imagine (I'm a vim user
myself and I find its true for me) that an emacs user has already invested in
the tool enough that learning a little more to make it even more useful is not
a big deal.

~~~
tikhonj
I can't help thinking that your average user is much less likely to write an
extension at all. Also, I suspect a disproportionate number of GNU program
users sufficiently invested in them to write extensions are either Emacs users
already (it is Gnu Emacs, after all) or at least fairly familiar with Lisp.

The sort of person who doesn't go out of his way to learn a new tool is also
the sort of person who probably doesn't go out of his way to _extend_ an
existing tool either. I think targeting advanced users for writing extensions
is completely reasonable.

Plenty of "average" (by your definition) programmers who use Eclipse know Java
well enough to write Eclispe plugins. How many of them do? I suspect a large
portion of those who do would have written the plugins in Guile as well.

~~~
dkersten
Probably not all that relevant here, but I think that Java's verbosity is a
barrier to entry in itself, even if Java programmers do not think so. When I
worked in a Java shop a few years ago I would have happily coded up quick
extensions and scripts[1] in something like python, but never even considered
doing so in Java because there was too much effort involved. I guess I'm
arguing that the users must feel that the benefit of writing a script or
extension must feel like its much greater than the effort of not doing so
_and_ the effort of writing the extension and that Guile is just plain too
unfamiliar to most that the effort _seems_ to outweigh the benefits, even if
in reality this is not true.

[1] I did actually write a few Python scripts to generate repetitive Java code
because I felt that doing so would be much less effort than writing the code
by hand.

------
jwatte
Quote: " Scheme's macros have no comparison in any other language."

I argue that Erlang parse transforms are at least as powerful. Interesting
that he mentions Erlang, but doesn't mention that! (Erlang probably isn't
ideal as an /extension/ language for other reasons, though)

~~~
zaphar
erlangs parse transforms are indeed just as powerful but no where near as easy
to use. They allow you to modify your source file in pretty much the same way
but the ast you are modifying is way more complicated than lisps and much more
difficult to properly modify.

If Lisps macros are difficult for the average user to learn then Erlangs Parse
Transforms are even more so and a much larger barrier to usage than lisps.
It's rare for an erlang developer to reach for a Parse Transform it's rare for
a lisp developer not to reach for one.

