
A Programmable Programming Language - samth
https://cacm.acm.org/magazines/2018/3/225475-a-programmable-programming-language/fulltext
======
hyperion2010
I have been building a new not-exactly-a-programming-language using Racket and
it has been an absolute pleasure, and the community has been extremely
friendly and helpful.

One existing stumbling block that I encountered on the way is that I could not
find documentation for the full process for the setup for creating a new
packaged language (I have been known to fail hard at finding the right
documents so this may just be my problem).
[https://beautifulracket.com/](https://beautifulracket.com/) has excellent
coverage of the general process, but the differences between #lang br and
#lang racket/base are just large enough that I couldn't fill in the gaps
without going and looking through multiple repos to see how others had
implemented their repo layout and then could intuit which setup commands I
needed to run.

If I find time I intend to write up my findings and submit them to the
documentation, the short version is here in case someone finds it useful.

    
    
      repo-base
      ├── .git
      ├── my-lang       ; run `raco pkg install` here
      │   └── info.rkt  ; meta package depends on my-lang-lib
      └── my-lang-lib   ; run `raco pkg install` here
          ├── info.rkt  ; include actual dependencies
          └── my-lang   ; implementation goes here
              └── main.rkt  ; in theory you can put your whole implementation here
    

Once that set up is complete you should be able to use `#lang my-lang`.

~~~
araes
Thank you for the link. Reading the example on the creation of stacker.rkt (a
toy reverse polish language) was dramatically more enlightening than the main
article. As I understand, Racket = source-to-source interpreter with a lot of
helpful shorthand for defining syntax of {insert language here} or arbitrary
language you make on the spot. Do they already have libraries for major
languages like c++, JS, python? Would be nice to write in a syntax I like
(perl) and turn it into c++ that compiled.

~~~
hyperion2010
Racket isn't really a source to source compiler/interpreter (you could try to
write a compiler that converted racket into your language of choice, but that
can often be quite difficult). Everything ultimately has to pass through
Racket's language semantics, so matching implementation details requires a lot
of work if the underlying language spec is large. The #lang C example down
below sidesteps these issues because it acts as a pass through to the system c
compiler and then handles returning those results into racket, essentially it
delegates the language semantics to the c compiler and only deals with how to
map the results of calling c code back onto racket objects. Similar issue with
the FFI if you want to translate from one language to another, you still have
to go through the process of mapping language semantics.

Two examples where languages/parsers have been implemented in racket:

Algol60 is the best and most complete example [https://docs.racket-
lang.org/algol60/index.html](https://docs.racket-lang.org/algol60/index.html).
A python parser has been implemented
[https://github.com/dyoo/ragg/blob/master/ragg/examples/pytho...](https://github.com/dyoo/ragg/blob/master/ragg/examples/python-
grammar.rkt)

~~~
araes
Thank you for the clarification. Even after two articles I obviously still
didn't fully get it. Appreciate the help, and the example links.

~~~
fiddlerwoaroof
Racket uses macros to transform your language’s syntax into valid racket code,
which is then interpreted.

~~~
skrishnamurthi
No, this is far too glib, and the last part is wrong. See
[https://www.hashcollision.org/brainfudge/](https://www.hashcollision.org/brainfudge/)
instead.

------
QuadrupleA
I don't think I've seen any convincing examples yet of how writing a new
language or DSL is superior to the already-very-flexible idea of user-defined
functions, supported by almost all languages. Isn't a DSL just syntactic sugar
in most cases?

Functions are verbs, usually named in problem domain terms, that can operate
on a huge variety of nouns (data) also namable in the problem domain. So I
don't buy that existing languages can't represent the problem domain, or force
programmers to only do things in machine or traditional-language terms.

Don't want to be a spoilsport either, glad people are researching and
experimenting with new approaches - just not sure I'm convinced yet the "make
a DSL for everything" idea seems all that promising or different from what we
already do as programmers.

~~~
da_chicken
> Isn't a DSL just syntactic sugar in most cases?

I don't think so. Maybe with things like LINQ, but in many cases a DSL lets
you abstract things to such an extent that you get a new way of thinking about
the problem. That's why DSLs are almost always very high level languages.

SQL and regex are the obvious counterexamples, where you don't describe _how_
to return the data you're looking for or how to store it on disk, you just
describe that data and the language does the work for you. That's a completely
different way of thinking about a problem compared to procedural logic. The
languages allow you to describe very complex data in an explicit and unique
manner, and in doing so allow you to abstract what the data is from how the
data is maintained. Obviously, that can bite you if you structure your DB
poorly, but on the flip side it allows people to learn how to use very complex
descriptions of what are essentially abstract concepts instead of needing to
know details that are extremely easy to screw up. Declarative programming is
extremely powerful because of this, and it allows for extremely complex
systems to be built on top of this new layer of abstraction.

A shell is another counterexample where being "just" semantic sugar is really
oversimplifying it. You're in a fixed context with a default function, so you
get a lot of acceleration over having to spawn all those processes by hand.
You're stripping out so much overhead code that you're into a new level of
abstraction where -- unless you need to -- you don't think in terms of
procedural logic.

For a less straightforward answer, a generic programming language would be
like a hex editor, whereas a DSL would be like PhotoShop (or even Paint).
Sure, a hex editor can modify any type of file in any way, but the power of
PhotoShop comes from all the power you get out of being domain specific. It's
not just the additional built in functions the program gives you, it's also
the manner in which the data are presented that allows you to see things which
you simply would miss at the hex editor level. Here the abstraction is
allowing you to see what the data actually represent instead of just seeing
how the data exist in the computer.

~~~
kerkeslager
Let's take one of your "obvious counterexamples":

Exhibit A:

    
    
        [A-Za-z]\w*\(\d+(,\d+)*\)
    

Exhibit B:

    
    
        cat(or(range("A","Z"), range("a", "z")), zero_or_more(alphanum()), "(", one_or_more(digit()), zero_or_more(cat(",", one_or_more(digit()))), ")")
    

Exhibit A is _absolutely_ syntactic sugar for Exhibit B.

Regex is great if someone has already implemented a regex engine for you. But
a key part of the post you're responding to is that it's talking about writing
a DSL yourself.

I can write the functions to run Exhibit B in probably 3 hours. I'd challenge
you to write a reasonably language-integrated Regex engine in under 3 _days_ ,
even assuming you have a good understanding of the domain. That difference in
implementation time is going to take a while to pay off in composing actual
regular expressions. And regular expressions are a _very simple_ DSL. A more
complex DSL would take longer.

~~~
Bjartr
I think, taken to the extreme, all programming languages would then "just" be
syntactic sugar over lambda calculus

~~~
mtreis86
Greenspun's tenth rule: Any sufficiently complicated C or Fortran program
contains an ad-hoc, informally-specified, bug-ridden, slow implementation of
half of Common Lisp.

------
apeace
I recall going to a talk when I was at Northeastern, and at one point the
speaker said something like, "But why would you create a new programming
language just to solve one problem?"

Matthias Felleisen (the guy in the video) interrupted him from the audience to
respond, "I know several people who have done exactly that, and it worked out
quite well for them!"

As he says in the video, Northeastern teaches students how to program in a
systematic way, rather than by copy-paste-modify from other examples. They
certainly teach a powerful way of thinking, and it has served me well over the
years.

If anyone out there is reading this and is thinking about going to college for
computer science, definitely check out Northeastern. I could not be more
thankful that I went there.

You can read the freshman text book online: "How to Design Programs"[1].

[1] [http://htdp.org/](http://htdp.org/)

~~~
discreteevent
The problem is that the several people he knows are not the average
architecture astronaut. It takes a lot of taste and experience to design a
usable language.

~~~
humanrebar
And it takes a lot of _organizational_ dedication and work to support the
language properly: training, documentation, packaging, regression testing (1),
editor support, debugger support, libraries, and the list goes on. I doubt a
single individual could do all that for more than a short while, so you need
an organization that wants to do all that right for the sake of the language.
Most companies won't really care that your language isn't in fedora upstream.

(1) Lots of popular languages get the community to do their regression testing
for them, but if you are the sole user of your language, you need to do all
the testing yourself somehow.

------
gwbas1c
I can just imagine the learning curve when encountering legacy code in a
language that's only used for the project, product, or company. I can't begin
to explain how many times I have just Googled a question in a particular
programming language and very quickly found an answer. When I work in new or
unpopular languages, I often can't find an easy answer.

The difference in time is staggering. The learning curve for a popular
language means that learning something new takes me 5 minutes, but in an
unknown language learning something you could take me all day if there are not
a lot of things like stack Overflow posts.

If a project specific custom language deviates too far from the norms, then it
means that incoming programmers will have a harder time maintaining it.

~~~
Too
Having seen all kinds of abuse of C macros or python metaprogramming I can't
even imagine the mess this could create. Some people just seem to love the
idea of creating a undebuggable Frankenstein macro system just so they can
save 3 characters of code when writing foo.bar instead of foo["bar"]. Python
libraries depending on @ annotations everywhere is another favourite. You save
1 measly line of code by not having to call or register the function somewhere
but you loose so much more as you can't control initialization order and your
program usually is forced to store it's data as global state.

I do see the powers and benefits this can give but with great power comes
great responsibility. It must be truly isolated to places where you absolutely
need it instead of just for fun.

~~~
heavenlyblue
Oh my god - this. I have recently started writing my own library that required
to register functions for the task runner. At first I’ve tried to copy the @
approach, but then realised what an unmaintanable mess that becomes.

------
montrose
"Lisp is a programmable programming language." — John Foderaro, CACM,
September 1991.

~~~
matthias_f
Yes our title intentionally alludes to CL.

------
yogthos
Instead of having to think about every possible way a language might be used
and account for it up front, languages with good metaprogramming facilities
let the users them to fit their needs. This way the core language can stay
small and focused while remaining flexible.

While you don’t want everybody building their own language, userland
extensibility can play a huge impact on the overall cleanliness of the
language because new ideas can be tried out as libraries. If these ideas don’t
pan out, the community moves on. Projects using these libraries will keep
working, but the core language doesn’t end up accumulating cruft. I wrote
about that in detail here:
[https://yogthos.net/posts/2017-10-03-MovingForwardByLettingG...](https://yogthos.net/posts/2017-10-03-MovingForwardByLettingGo.html)

~~~
mcguire
...which is why Scheme has, like, 37 different, incompatible object systems.

Everything has costs.

~~~
eadmund
That's Scheme. Lisp, meanwhile, has one: CLOS. One is excellent for learning
about things like hand-rolled object systems, and one is excellent for
building large systems.

~~~
iak8god
_Common Lisp_ has CLOS. I'm aware some purists don't think Scheme is a Lisp
but, regardless of one's stance on this, there are most definitely other Lisps
than Common Lisp.

~~~
YouAreGreat
Having called the result of Lisp standardization _ANSI Common Lisp_ instead of
just _ANSI Lisp_ doesn't change the fact that the whole effort was all about
standardizing Lisp.

Moreover, why not just say Scheme or Clojure when talking about Scheme or
Clojure?

If ANSI C had been called ANSI Common C, would Java fans never cease to insist
on calling Java a dialect of C?

~~~
iak8god
> the whole effort was all about standardizing Lisp

There are more Lisps now than ever, so perhaps that was not as successful as
was hoped.

> why not just say Scheme or Clojure when talking about Scheme or Clojure?

Why not just say "Common Lisp" when you mean Common Lisp, and "Lisp" when
making true general statements that include other things that have always been
considered Lisps? How are we to refer to Emacs Lisp without tying ourselves in
knots if we refuse to say that "Lisp" includes it?

------
mark_l_watson
Great article. I have in the past (when I wanted to use Scheme) used either
Gambit (great support from the author!) or Chez Scheme (fast!), and only have
played around with Racket a bit (perhaps 50 hours in the last 5 years).

I have been thinking of switching to Racket, and this article 'pushed me over
the edge' on that decision.

------
nickpsecurity
Language-oriented programming sounds like metaprogramming with DSL’s just with
a new toolkit. Language-oriented programming might be a more approachable term
for that, though. If I heard it, the first things I’d think of were tools such
as Rascal and Ometa that let one arbitrarily define, operate on, or use
languages. That covers language part. Far as integration, a number of techs
supporting DSL’s… probably a lot of LISP’s and REBOL… had a powerful language
underneath one could drop into.

So, this seems like a new instance of old ideas they’re giving a new name. I
do like how they’ve integrated the concept into a general-purpose, GUI-based,
programming environment instead of it being it’s own thing like the others.
You get best of both worlds.

An old idea I had was researchers should do more experiments in building
things like Rascal or Ometa alternatives in Racket leveraging what they
already have to see how far one can stretch and improve it.

[http://www.rascal-mpl.org/](http://www.rascal-mpl.org/)

[https://en.wikipedia.org/wiki/OMeta](https://en.wikipedia.org/wiki/OMeta)

~~~
gfodor
Language oriented programming is far from a new term. See the history of
jetbrains MPS and intentional software. There was a strong meme in the early
2000's this was the future, it never seemed to take off.

~~~
username223
I even bought into that meme a bit in the early 2000s. It turns out that most
"language" designers (including myself) aren't that good at it, and that
designing a language that isn't just yet another leaky abstraction takes a lot
of work. Your "interface" is the entire syntax and semantics of your
"language."

~~~
stevedonovan
Yes - there is definitely a hierarchy of difficulty: user code, library,
macros, full syntax extension. The trouble is that designing new languages
takes a lot of _taste_ and experience. We don't know the full domain, so DSLs
tend to have bits added to them as knowledge of the domain grows - and that's
the scary point in language evolution. So, despite these things being
tremendous fun, I tend more towards correctness and clarity at the expense of
sheer expressiveness. (An example of ad-hoc language design is CMake, which
makes my eyes bleed. Kitware should have used an existing language - even tcl
would have been better.)

------
cornholio
I wonder how this approach scales to large code bases maintained by global
teams. Instead of the usual discovery that each contractor or "rockstar"
programmer created modules using their preferred language or in vogue
technology, you might now find they derived their own obscure language flavors
that read right to left and use "=" for function calls.

In large projects, readability and maintainability trump expressibility.

~~~
cat199
> In large projects, readability and maintainability trump expressibility.

and neither is a substitute for proper governance...

------
zitterbewegung
Racket is awesome. I originally used scheme48 but was impressed with rackets
tooling (when it was known as plt scheme) . After it branched out as racket
and added a large standard library with batteries included it became much more
practical to use. I am experimenting with using it with smart contracts and
having a library where you can make your own language and work with program
synthesis to web development it’s really awesome!

------
MaxBarraclough
> Racket eliminates the hard boundary between library and language, overcoming
> a seemingly intractable conflict

I don't buy it.

A language extension means the syntax wouldn't ever be valid in the base
language. The fact that Racket defines its 'for' construct using
macros/functions/whatever is cute, but I see no reason to pretend it's a
language extension.

~~~
shakna
> I don't buy it.

> A language extension means the syntax wouldn't ever be valid in the base
> language.

Sure.

    
    
         (struct pt ([x : Real] [y : Real]))
    

That's not valid Racket, but is valid Typed Racket.

A more extreme example:

    
    
        int main(int argc, char** argv) { return 0; }
    

That's C! Oh wait, it's the C #lang [0]...

Which means you can things like:

base.rkt:

    
    
        #lang C
    
        float f( float x1, float x2 ) {
          return x1 / x2;
        }
    

main.rkt:

    
    
        #lang racket
    
        (require base)
    
        f(10.2, 1.8)
    

Those syntaxes aren't compatible, by any stretch. The magic is, Racket gives
you access to powerful pre-processing, in the form of some amazing reader
macros.

What syntax you use, doesn't really matter, so long as you provide that
bridge. They're incompatible syntax, with a custom made parser allowing data
to flow both ways.

[0] [https://github.com/jeapostrophe/c](https://github.com/jeapostrophe/c)

~~~
mcguire
Shouldn't that second example be

    
    
        (f 10.2 1.8)

~~~
shakna
Yep. My bad.

------
skywhopper
I’m curious how this compares with the similar built in tools for building
grammars, etc in Perl 6, and also if the syntax flexibility possible is worth
the trade off vs using more native (if obscured or downplayed) syntax eg the
sorts of DSLs you can build with Ruby.

------
bsenftner
Wasn't "Script X" Apple's failed attempt at this back in the early 90's?
People found creating custom for every problem was unmaintainable. Each
developer was encouraged to customize their own personal language, and work
with syntax translators to work with group written code. Nightmare.

~~~
erikj
I presume you are talking about ScriptX created by an Apple spin-off Kaleida
Labs? I'm reading about it right now, and I don't see any specific features
useful for language-oriented programming, not even Lisp-like macros. From what
I can see it doesn't seem to encourage this approach to software development
at all, it looks like some kind of Smalltalk derivative with multiple
inheritance bolted on.

------
dwarman
Forgive me, but I recall, with some decades of practice, a major point in
FORTH is that one constructs a vocabulary and syntax specifically for solving
the problem at hand. Compiler primitives are just as first class as regular
word (aka function) definitions, including how to compile itself as well as
what to do when executed (DOES> keyword, which can itself be redefined). FORTH
may have a clunky and primitive reputation, but it seems to me it qualifies as
a programmable programming language.

~~~
skrishnamurthi
Sure. But the degree of clunkiness matters. Forth doesn't have the ability to
build linguistic _abstractions_ the way Racket does. But if you want to add
Forth to the evidence of the paper's thesis, that's fine, more the better.

------
criddell
This is offtopic, but seeing ACM made me think about my long-lapsed
membership.

Are you a member of the ACM? What do you get out of your membership?

------
trakam
A side note: React and jQuery are not really complementary libraries dealing
with separate concerns..as suggested by the article,unless i have
fundamentally misunderstood both.

------
antirez
Tcl can do wonders in this area. _uplevel_ is in some way more powerful than
Lisp macros, since it happens at runtime.

~~~
ken
Isn't "uplevel" essentially Tcl's way to choose dynamic scope? Lisp has ways
to do that, too.

~~~
antirez
It's more like "eval" in the stack frame you select. This added to the fact
Tcl, like Lisp, has the same form for all the commands, allow to do anything
you could conceive and it will look like if it's part of the language. For
instance you can do a looping construct that works in a different way
depending on the calling function name or state or any other crazy stuff like
that.

------
samdg
Title should be edited: Lanugage -> Language.

~~~
sctb
Thanks! Updated.

------
tambourine_man
Typo in the headline: lanugage

