
The Emacs Problem (2005) - swah
https://sites.google.com/site/steveyegge2/the-emacs-problem
======
gruseom
I've noticed two interesting things about Emacs over the years. One is that
it's an order of magnitude easier to write tools in Emacs than in other
environments. It's also more satisfying, because Emacs lacks the arbitrary
hoops you typically have to jump through when writing plug-ins—hoops that come
from internal limitations of the host system and give you that annoying "why
can't I do it this way" feeling. Emacs, despite its high learning curve, has
little such systemic friction. Because making tools is an order of magnitude
easier and more satisfying, people do it more, which is why Emacs has so many
tools, as well as sophisticated things (e.g. Slime) that show up more rarely
elsewhere.

The other thing is that the core of Emacs is a masterful piece of domain-
driven design, where the domain is "text editing and windowing". Its
conceptual model is clear, simple, and consistent. (Some of the later
additions don't have this quality, but fortunately they mostly stay out of the
picture.) It has a rich language for its concepts—'buffers', 'text
properties', and so on—and uses that language everywhere through documentation
and code. This tremendously eases the burden of writing programs that
interoperate, because everything is based on the same conceptual model and
thus makes sense in the same way. Not all Emacs programs follow identical
conventions—far from it—yet it's amazing how close they get, given how many
there are. It's Emacs' exemplary domain model, as much as its Lisp character,
that creates this conceptual unity.

So while Emacs rightly gets a lot of credit for the technical aspects of its
extensibility, we should hear more about the design aspects. Its core design
is a masterpiece and ought to be studied as an example of the power of
software design itself—something we're mostly still pretty bad at. It might be
hard to get that taken seriously, though, since on the surface Emacs is
obtuse, clunky, and old. Only when you dive underneath does it become orderly
and beautiful.

------
patrickmay
"The whole nasty 'configuration' problem becomes incredibly more convenient in
the Lisp world. No more stanza files, apache-config, .properties files, XML
configuration files, Makefiles — all those lame, crappy, half-language
creatures that you wish were executable, or at least loaded directly into your
program without specialized processing. I know, I know — everyone raves about
the power of separating your code and your data. That's because they're using
languages that simply can't do a good job of representing data as code. But
it's what you really want, or all the creepy half-languages wouldn't all
evolve towards being Turing-complete, would they?"

This touches on the core problem with the whole Java ecosystem. The existence
of so many tools demonstrates, first, that the underlying technology is
insufficiently expressive and, second, that many programmers would rather
create tools than applications.

Leaving the second aside, the kinds of tools required for Java development are
largely unnecessary in the Lisp world because the language itself is
expressive and extensible. You don't need different representations for data
and code, separate parsers, and frankly bizarre build systems ( _cough_ maven
_cough_ ).

As ESR, I believe, said "Lisp is worth learning for the profound enlightenment
experience you will have when you finally get it." Unfortunately, it also
makes you more aware that things don't have to be as bad as they are in the
Java world.

Hopefully s-expressions will be the next aspect of Lisp rediscovered in
"modern" programming languages.

~~~
yummyfajitas
I have no difficulty with configuration. I have a reference.conf which
provides sensible defaults and is bundled in the jar file:

    
    
        cache {
          ttl = 1024
        }
    

I have an application.conf which belongs on the classpath and contains
location specific configuration:

    
    
        cache { 
          redis_server = "123.456.789.012:6379"
        } 
    

In my code, I do this:

    
    
        server = getRedisServer(config.getString("cache.redis_server"))
        server.put(key, value, ttl=config.getInt("cache.ttl")
    
    

[https://github.com/typesafehub/config](https://github.com/typesafehub/config)

Java does have fugly enterprisey frameworks. But it's silly to claim that this
is some property of Java.

Incidentally, Python also does "code as data as config" also - see django's
settings.py, for example.

~~~
patrickmay
You're correct that the particular tools and XML everywhere culture of the
Java ecosystem isn't a property of Java itself. However, the verbosity and
corresponding lack of expressiveness of Java does, I think, have an impact.

Elegance seems to be much more highly valued in the Lisp world than in the
Java world.

~~~
pjmlp
All the languages that get wide adoption at the enterprise level suffer from
similar issues.

I think it is something that is bound to the enterprise architect culture.

~~~
marcosdumay
Maybe it'd better to write that as: Only the languages that suffer from
similar issues get wide adoption at the enterprise level. (Yet, I don't think
that completely captures causality.)

Enterprise architects have a surprisingly big diversity of culture. I'd look
first at the process by what enterprises choose their development stacks. It
often has little input from architects, developers, testers, or admins.

~~~
pjmlp
My employer does enterprise consulting in Fortune 500 companies, the developer
stacks and architecture designs we sometimes end up doing, chosen by
architects, end up in what is usually called "space station architecture".

------
pmr_
Re-reading this after a few years was interesting. At the time it really
seemed like the Emacs ecosystem is stagnating but this has changed during the
last two years: package managers, crazy good, well maintained extensions
(magit, org-mode, flycheck, Gnus), and small language changes. We even had an
Emacs-only conference! ;)

Still, elisp is elisp is elisp and there is no getting around that fact. The
last Common Lisp rewrite/transformation has, AFAIK, stalled. 'cl is still not
welcome in core packages. Just staying with the language and dealing with its
quirks has proven very effective and I think this is in a large part due the
flexibility of being a Lisp. It might be the wrong language for text-
processing, but is the right language for ancient software that tries to stay
recent.

~~~
rcthompson
I agree, the "emacs problem" is more or less solved by the advent of ELPA,
along with external package repositories that mean you don't have get your
code into emacs to get it widely disseminated. Emacs Lisp is still Emacs Lisp,
but it has closures now, so who cares?

------
jpatte
_Most programmers these days are quite astonished that anyone would actually
still use Emacs. If they realized how much functionality it has, and how
powerful its extensibility model is, they 'd be ten times as astonished. It's
got things that Eclipse may never have, not in a hundred years, and that's no
exaggeration._

Thrill me.

~~~
agentultra
I've heard Emacs referred to as a Lisp run-time that happens to implement a
text editor within it. You can reprogram it within itself without stopping to
parse, recompile, or reload any code. This is pretty normal for Lisp-based
applications.

One interesting library I've come across is _skewer-mode_. It depends on a
package that implements an asynchronous web server to inject some handy
javascript into your browser that opens up a two-way channel with Emacs. I
just have to add the href to load the script in my file and then I can open up
a javascript REPL in emacs and load code into my browser without refreshing or
anything.

I've heard of projects that let you programmatically embed and control webkit
instances.

I publish books from emacs.

It's a handy _programming environment_ and not merely a single-purpose
application.

~~~
specialist
I would like to add that customizing/extending Eclipse has become a nightmare.
Two of my plugins stopped working in 4.x. Took forever to get a simple Nature
and Builder working again under 4.x.

Eclipse's uses XML to declare UI, behavior, expressions. Completely baffling.
Terrible docs. Extremely difficult to debug. I was reduced to trial and error
(after copying working examples I found).

I've used Eclipse since the early alphas. Its muscle memory.

Scouting the plugin dev stuff of IntelliJ, it doesn't appear to suck so
mightily. I've forced myself to start using IntelliJ.

I miss LISP so much I could cry. Even so, I remain wary of metaprogramming.
And I very much want to do plugin dev, vs modifying the runtime, for the
separation of concerns. I've had to customize all-in-one runtimes and borking
the whole runtime with a bad override is not fun to debug.

I've not done emacs plugins, so can't comment or compare.

~~~
agentultra
> I miss LISP so much I could cry. Even so, I remain wary of metaprogramming.
> And I very much want to do plugin dev, vs modifying the runtime, for the
> separation of concerns. I've had to customize all-in-one runtimes and
> borking the whole runtime with a bad override is not fun to debug.

Your concerns are artificial! Let go of your concerns and be free. The
hundreds of Emacs users around the world are already doing it. It's great.

It used to be a bit of a wild-west without lexical binding but we have that
now and it's going over pretty well. Fewer libraries step on each others' toes
as adoption spreads.

We have package management built-in. We have awesome introspection and
debugging tools. There are a lot of wonderful built-in packages.

My init file isn't even written in raw elisp anymore. I use a well-documented
org file. The amazing org-babel library is bundled in. My init file is written
for humans to understand and only incidentally for my editor to execute. It's
great.

I only wish more software was designed this way.

~~~
_nh
> My init file isn't even written in raw elisp anymore. I use a well-
> documented org file. The amazing org-babel library is bundled in. My init
> file is written for humans to understand and only incidentally for my editor
> to execute. It's great.

Came here to say that I do this, too:
[https://github.com/nhoffman/.emacs.d](https://github.com/nhoffman/.emacs.d)

Not sure if it was worth the effort, but it was fun!

~~~
jsw97
I do this too. I got the idea from Sacha Chua:

[http://sachachua.com/blog/2012/06/literate-programming-
emacs...](http://sachachua.com/blog/2012/06/literate-programming-emacs-
configuration-file/)

------
jerf
Before someone jumps up and says "JSON!", bear in mind that while it's often a
better format for moving data, and that the stacktrace example in that post
would probably be better as JSON, that JSON does not solve the arbitrary text
problem. While I'll take JSON in a heartbeat for most data serialization
tasks, if you've really got _mixed text_ , I'll take XML.

~~~
sipior
Could I trouble you to elaborate on that a bit? I'm not sure I understand why
XML would be better than JSON in that case.

~~~
chubot
I think Steve Yegge (in another post) said it surprisingly well, and simply.
Paraphrasing:

If you have more data than text, use JSON. If you have more text than data,
use XML/HTML.

Imagine HTML documents encoded in JSON -- in that case JSON would be ugly and
annoying to edit. Now imagine (the more common situation of) structured data
serialized as XML. That is ugly and annoying to edit.

Use the right tool for the right job. :)

~~~
ygra
The two are not interchangeable, though. XML allows mixed content, i.e.
something like

    
    
        <Foo>Some text <Bar>more text</Bar></Foo>
    

while JSON does not. So there isn't a 1:1 mapping, at least not in the
direction XML → JSON (and the other direction only works if you have a schema
that defines data types for the document).

~~~
ingenter
One _can_ make a 1:1 mapping from XML to JSON, but it is going to be verbose.

    
    
        {
           'tag': 'Foo',
           'children': [
               'Some text',
               {
                   'tag': 'Bar',
                   'children': ['more text']
               }
           ]
        }
    

or (hello, LISP)

    
    
        ['Foo', ['Some text', ['Bar', ['more text']]]]

~~~
jerf
Yes, that's what I mean. Both XML and JSON can ultimately encode everything,
but they both do indeed have strengths over the other. The problem with XML
isn't that it is a "bad thing" that should never be used, the problem is that
it become trendy and people started using it for all sorts of things it
shouldn't have been, combined with a healthy dose of it being used by a lot of
people who didn't even understand XML. (XML isn't _that_ complicated, but
there's more to it that "tags, attributes, and text".) When you're in a domain
where XML really should be used, you're going to regret trying to jam JSON
into that domain.

Also, side observation, that Lisp-like encoding is pretty icky as-is (deciding
if something is a tag or text by how deep the array is?), and gets worse if
the XML uses attributes.

~~~
tel
I think the various XML Schema-ifications don't help either. XML is fairly
poorly defined while JSON is pretty precise. This has more to do with a heavy
industry of poorly defined, implied semantics atop XML than anything about the
raw format, though.

------
zimbatm
It would be interesting to see lisp adoption coming not as a language but as a
serialization format. It seems fairly easy to implement and every language
would then have at least the AST part of lisp.

~~~
wtbob
You mean like canonical s-expressions
([http://people.csail.mit.edu/rivest/Sexp.txt](http://people.csail.mit.edu/rivest/Sexp.txt))?

One can write a parser for the canonical representation in less than an hour,
and a parser for the advanced representation in a weekend.

They are infinitely nicer than XML.

~~~
zimbatm
Yes. So why isn't it more common ?

------
blueblob
I find the intro paragraph kind of funny that "Lisp sure beats the hell out of
Java for text processing". The reason that I find it funny is that The
Stanford Parser/Stanford CoreNLP library is one of the more popular toolkits.
Perhaps the status of Lisp text processing is such that it is not needed but
Wikipedia has only one Lisp toolkit and quite a few Java ones[1] (Ruby also
has 1).

The other thing that I found interesting in this article was the reference to
regular expressions as being really useful for text processing but not good
for handling tree-based data and I very recently found that the Stanford
Parser has tree regular expressions/tregex[2].

[1]
[http://en.wikipedia.org/wiki/Outline_of_natural_language_pro...](http://en.wikipedia.org/wiki/Outline_of_natural_language_processing#Natural_language_processing_toolkits)
[2]
[http://nlp.stanford.edu/software/tregex.shtml](http://nlp.stanford.edu/software/tregex.shtml)

~~~
webkike
Simply by having a parser library - and I should mention one meant primarily
for natural language parsing - is not indicative of Java being "better" for
text processing than lisp. Indeed, one could argue that the mere fact that
Java has a library for text processing at all is indicative of lisp being
better; after all, the reason lisp does not have an external package could be
because it simply does not need one to be just as powerful.

------
arca_vorago
I have been learning Emacs for a few months now, and am loving it (mostly),
but I'm going to skip the Emacs issue and focus on his problem with xml.

"Doing this effectively requires your data to be tree-structured."

I have actually been replacing XML with YAML in lots of places. I find the
structure much less verbose, and easier to understand, and has features that
JSON lacks, and I almost never have problems with delimiter collision. If you
haven't yet, I really suggest checking out YAML, but that's just my two cents.

~~~
mst
You may also like [http://p3rl.org/JSONY](http://p3rl.org/JSONY) which ingy
and I came up with as a sort of half way house between YAML and JSON. Note
that while that's a perl implementation, the Pegex grammar underlying it
should be compilable in other languages.

------
badman_ting
I wish I could see it compared to something more like Sublime Text. Of course
Eclipse & Visual Studio are crappy, we all know that. Nobody is questioning
that.

Also, "plugins are super-easy to write" is not much of a benefit to someone
who wants to, you know, actually write apps rather than write plugins for the
editor in which they will ostensibly one day write an app. Imagine if someone
told you to use app X instead of Photoshop because the plugins are easier to
write than Photoshop plugins.

~~~
coldtea
> _Of course Eclipse & Visual Studio are crappy, we all know that. Nobody is
> questioning that._

You'd be surprised. They might have clunky editors, but they're far more
advanced in dealing with code intelligently (refactoring, AST parsing and
transformations, knowledge of structure etc) than ST or Emacs.

~~~
badman_ting
In my experience with VS that stuff mostly came from Resharper, a plugin.

------
TheLoneWolfling
> you have a third option [...] you store your text data as a lisp program.

And now anyone with access to a "text" file can run arbitrary code. Or am I
missing something?

~~~
nnq
You are missing that by "reading" the code in Lisp you don't also "execute
it", you can just read it and have a nice tree data structure. If you have any
reason to eval it, go ahead, but that would be a bad idea for a log or a
config file (unless you want a Turing complete configuration or logging DSL...
hopefully you know this is a bad idea!)

Most other languages don't have an obvious way to just parse a file of source
code without actually executing it: `#include <stdio.h>` or `import
mycoolpythonlib` or `require __DIR__ . '/goolib.php'` both _parse_ and
_execute /compile/link_ the file, so yes, any config/log file becomes a code
file and "arbitrary code" gets executed, but it doesn't have to be this way.

(Now, you have things like "import hooks" and access to the AST in Python for
example if you want a tree of the imported file, but nothing "simple" and
obvious. And this is the thing actually: as much as people consider Lisps too
complex and powerful, the nice thing is how "obvious" and "simple" they make
things like this - by just parsing sexp tree and manioulating nested lists
instead of inventing a file format like XML with its transformation languages,
query languages and so on, and all the tools to support them, and all the man-
years (or is it decades? ...centuries?) spend to code and maintain them
...just think how much needless effort could have been spared.)

~~~
saw-lau
Now you've kick-started me to start looking at 'Land of LISP' which has been
sitting on my shelf for a number of years...

~~~
nnq
Glad to have resparked the interest. But as cute and witty its comics may be,
I couldn't stand Land of LISP and abandoned it after ~25%... not "dense"
enough for me I guess.

Practical Common Lisp is what grabbed me finally (it's available for free
online [http://www.gigamonkeys.com/book/](http://www.gigamonkeys.com/book/) or
you can buy the print), and you'll love it if you more "to the point" and
practical examples.

------
_pmf_
> Unless "we" are C++ programmers, in which case "we" like to write 2500-line
> clones of the Unix 'grep' utility whenever "we" need to do a text search —
> or so it would seem, from the candidates I've interviewed in the past few
> months.

Maybe you made the impression of a smart ass interviewer who likes to make up
retarded interview exercises, and your interviewees played along. You might
want to work on that.

~~~
GFK_of_xmaspast
That's probably a pretty shitty interview question, but can it with the
"retarded" stuff ok.

~~~
AnimalMuppet
Yeah. The assumption of smug superiority on the part of the Lisp folks can be
really annoying, but responding in this way just makes you also a jerk. Been
there, done that. Just yesterday, in fact. After thinking about it, I decided
that I didn't like the way I'd gone about it. It wasn't that the other person
was being wonderful - he/she was being a pompous, self-important jerk. But I
let that turn _me_ into a jerk, and that's not progress...

------
AnimalMuppet
Down in the comments, he says that Lisp isn't really ready for production use.
He lists some reasons. This was written in 2005; does anyone care to comment
on the current situation?

~~~
wglb
It is performant (I use SBCL), libraries are just about trivial to use now
that zach did quicklisp:
[http://www.quicklisp.org/](http://www.quicklisp.org/). I use it for almost
everything that I do now. Even pen testing.

