Hacker News new | comments | ask | show | jobs | submit login
Show HN: Compiler using Lisp’s macro system for metaprogramming C-like languages (github.com)
80 points by dis0rdr on Jan 5, 2018 | hide | past | web | favorite | 32 comments

Reminds me of an older (unfinished) project of mine, sxc:


I don't understand, what are lisp/scheme/racket/haskell being used for? I mean how are those languages good from a software engineering point of view? From a project management point of view?

Is it possible to hear bad things about those languages? Why aren't they used? Is it because they are too hard to use, or to difficult to approach for a beginner?

I can speak for CommonLisp, scheme and Racket. Someone else will have to answer for Haskell. Generally, LISPs are extremely flexible, reasonably fast, compiled languages with macro systems that allow you to develop a domain specific dialect. Some of them, like CommonLisp and Racket, come with a pretty huge set of libraries, tooling and ecosystem.

> Why aren't they used?

They are used a lot.

> Is it because they are too hard to use, or to difficult to approach for a beginner?

They tend to be easy to learn but difficult to master for writing idiomatic code, i.e., have a relatively long flat learning curve.

> Is it possible to hear bad things about those languages?

Sure. In my personal experience, their biggest downside is the same as their biggest selling point: their flexibility. Code is generally hard to maintain and read by others. You can invent some incredibly hacky spaghetti code monstrosities in LISPs if you want to. More disadvantages: high memory use, dynamic typing and dynamic dispatch can suck big time (e.g. Racket OOP methods are mostly checked at runtime, which can lead to testing nightmares if you're not careful)

Overall, CommonLisp and Racket are pretty good 'batteries included' languages, whereas some scheme implementations are extremely fast and evolved small extension and scripting languages.

Theyre similar to Perl in the sense that theyre extremely powerful and flexible but they tend to be highly customized to the business domain of their design.

This type of languages enable a very small team of engineers to be profoundly productive but the learning curve with adopting an existing project can be quite high even for expert programmers due to the lack of consistent patterns you would expect for more Enterprise languages like Java .net or python.

The end result is great success for small smart teams but lack of interchangeability and scalability for Enterprises

Edit: I once wrote a closure web application for a volunteer project. The experience was wonderful and painless however finding Developers to a system maintain the code base remains a serious challenge today

As an industry we dont take maintainability very seriously. You think it would be easy to maintain web apps written in more "mainstream" languages from a couple of years ago?

The web industry constantly chases fads, sure, but that's not true of the whole software world.

Haskell and Scheme may be as far from each other as C and Javascript. You seem to suggest they are all the same thing, when in fact the most prominent thing they share is lack of mainstream (That is, a la Java mainstream) usage.

Just by the fact the a language is well engineered doesn't make it a popular language in the industry. The time and the way in which it appears makes a difference. When C appeared (from Bell labs) it fit nicely with the Unix ecosystem. There was a better engineered language already here, PLM, but the industry couldn't take and run with it as easily. The time point of Perl 5 successes is another example (and contrast that with the appearance of Perl6 more recently). Java, when it was introduced, was not well engineered at all but it came at a certain point in object orientation that the software engineering community had been striving toward.

Haskell and friends (Elm, ..) are fantastic, and perhaps the absolute best programming languages for software engineering. Except there's a small problem: Haskell is either write-only or read-only, depending on what you're looking at. That is, beautiful Haskell code is easy enough to read, but very very difficult to write, while working code you might write may not be very readable at all. Of course, that can happen in any language, but it seems to me that Haskell leads to that sort of situation very easily. On the other hand, the nice thing about Haskell is that once your code compiles it also probably does what you want it to :)

I would stay away from dynamically-typed languages (e.g., the Lisp and Scheme families) for software engineering. Dynamic typing means more run-time errors, which means a higher support burden, which is very much what you want to lower when you're a software engineer. And this is why I like Haskell: it's statically-typed. (The main appeal of Lisp is its macro system, really. Haskell gives you the sort of power that the Lisp macro system gives you anyways, though at the cost of more compile-time processing.)

If you can't use Haskell, then your best bets are Rust and C/C++.

In any case, a software engineer almost necessarily has to be familiar with all of these, and able to use any of them. You really want a strong foundation at the lowest layers (C, and even ASM) in order to work the full stack (libraries, applications, compilers, OS). You don't have to be a full stack engineer, naturally, but it sure helps to be able to adapt to working at different layers, and for this you need a strong conceptual foundation. A lot of layers in a full stack are written in C/C++, but web front-ends involve JavaScript, which is dynamically-typed, so you'll really need to be familiar with all of these, and that's just to get started.

> I would stay away from dynamically-typed languages (e.g., the Lisp and Scheme families) for software engineering. Dynamic typing means more run-time errors

Note that Common Lisp has type declarations, which can move type errors from run-time to compile-time.

ML is a nice language family if you want something which is typed and participates in the Lisp mindset.

Yes, in principle you can have a language that lets you cover the gamut from statically- to dynamically-typed code. Indeed, Haskell is such a language...

But it's not just whether the language supports it, but also:

- what is the default - what is the cultural default - the extent of type inferencing - the extent to which the compiler can generate efficient code (i.e., pass around unboxed, naked values sans run-time typing information)

CL basically doesn't go far enough. To begin with, the default is dynamic typing, and IIRC it has no type inference, though at least CL compilers can do a fair bit optimization but you'll always pay the price of some type encoding in pointer/fixed-sized-integer values' low order bits.

> Note that Common Lisp has type declarations, which can move type errors from run-time to compile-time.

This is a dangerous assumption. The standard does not require that Common Lisp type declarations cause the compiler to detect type inconsistencies (although some implementations might be smart enough to do so in some cases). CL type declarations tell the compiler it can remove runtime checks. They are performance optimizations. If anything they decrease type safety.

> CL type declarations tell the compiler it can remove runtime checks.

That's not true. To tell the compiler that it can remove runtime checks you declare the optimization quality SAFETY to 0.

The Common Lisp standard does not specify what type declarations do and what the interpreter or compiler does with it.

Some compilers will never check types. Some will use them when SAFETY is low as assertions and remove runtime checks. Some will use them as assertions both at compile and runtime.

> They are performance optimizations. If anything they decrease type safety.

That depends on the implementation and the compiler switches.

In SBCL by default it INCREASES type safety:

  * (defun foo (a) (declare (type (integer 0 100) a)) (+ a 10))

  * (foo -1)

  debugger invoked on a TYPE-ERROR in thread
  #<THREAD "main thread" RUNNING {1001950083}>:
    The value
    is not of type
      (MOD 101)
    when binding A

  Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

  restarts (invokable by number or by possibly-abbreviated name):
    0: [ABORT] Exit debugger, returning to top level.

  (FOO -1) [external]
               (DECLARE (TYPE (INTEGER 0 100) A))
               (BLOCK FOO (+ A 10)))

You're right. The standard leaves the interpretation of type declarations in CL up to the implementation. I was coming from a CCL perspective, and I should have checked the others.


  ? (defun foo (a) (declare (type (integer 0 100) a)) (+ a 10))
  ? (foo -1)
It's dangerous to assume the effect type declarations will have in CL. You have to test it.

Even in CCL it depends on the compiler settings:

  ? (declaim (optimize (safety 3)))
  ? (defun foo (a) (declare (type (integer 0 100) a)) (+ a 10))
  ? (foo "bar")
  > Error: The value "bar" is not of the expected type (MOD 101).
  > While executing: FOO, in process listener(1).
  > Type :POP to abort, :R for a list of available restarts.
  > Type :? for other options.
  1 > 
Type declaration added -> runtime safety increased...

Best read the implementation manual to see how it deals with optimization values and type declarations.

Apparently the solution is to check the type before you declare it: https://stackoverflow.com/questions/32321054/using-declare-t...

I assume that most CL compilers are smart enough to optimize calls to such checking functions when they occur in a context where the type has already been declared. And you can probably write a macro to make it more convenient ...

There's no guarantee any given CL compiler will optimize such calls away. It would probably be considered wrong for the compiler to optimize them away. Remember that type declarations mean "don't check the type, ever." They do not mean "check the type at compile time" because CL never checks types at compile time.[1]

The technique in the example can defeat the purpose in simple cases (like the example) because type declarations removed runtime type checking and then you manually added it back in.

But it's a useful technique in cases such as an inner loop from which you remove type checking inside the loop, but move it out of the loop. This is an advanced CL technique. You have to be very careful about such things as adding 1 to a fixnum in the loop and possibly overflowing it, which is exactly what you have to care about in C.

[1] modulo certain compiler quirks and the use of compiler macros, which are an advanced technique.

> CL never checks types at compile time

SBCL, CMUCL, Scieneer CL do.


You can enforce type checks by writing your own.

LISP is rather extensible. People have written Haskell-like type systems for LISP with compile-time checks.

My opinion: If you can't use Haskell, you should definitely use a null-free language for safety in this day and age, which means Rust and definitely not C/C++.

Let me know when I can use Rust in Android, iOS and UWP with the same level of tooling and libraries support as C++.

I want to eventually use it, but not at the price of my productivity.

Yes, but that doesn't mean that you shouldn't be able to use C/C++ as needed -- sometimes you have no choice.

As always, pragmatism is good. If you are good at C/C++, use it and be happy and productive. I presumed the context of this discussion, however, was wanting to use Haskell for its particular benefits, and in my opinion the null-free nature is a key benefit, which would not be achieved by using C/C++.

> I mean how are those languages good from a software engineering point of view?

Common lisp: great for game programming or any type of application where it takes a while to build up a set of "state." You can redefine various aspects of the program while it's running. So, ok, there was a bug in your physics code and your player fell through the map. You can fix the physics bug while the game is still running and place your player back in the same spot. Now you don't have to recompile and try to get your game in the state it was in before.

When I get back into game programming, I'm probably going to use CL extensively. I might delegate the low-level stuff to an embeddable game engine (like Orx) but all my high-level logic would be CL.

Are you asking why people don’t use functional programming languages? If so, I’d say it’s the same reason most people don’t use procedural programming languages: because multi-paradigm programming languages (that combine procedural and functional features) give us both with minimal cost. :)

I think the title might be misleading here. It doesn't really transpile Common Lisp to C, it requires you to write in a C-like Lisp dialect so it gets translated to C. Might as well have been an entire Lisp dialect with a compiler of its own.

I think the idea is that you write C-with-sexpr-syntax, but by embedding those s-expressions in Common Lisp, you can make use of macros written in Lisp to generate C code. Essentially, it's a saner preprocessor for C that requires you to write in funky syntax.

I think this would be even more interesting if it could do the C -> s-expression transformation, so that code already written in C could integrate calls to Lisp macros without needing a full rewrite.

A C->S-expression compiler would have to produce very obvious and simple mappings in order for a macro system to be usable. Another way of saying this is that a simple and standard AST would be nice.

The Vacietis (https://github.com/vsedach/Vacietis/) reader can be easily adapted to do that. You can see examples in the unit tests:


As-is, C block constructs get mapped directly to Common Lisp special forms like tagbody and prog because those implement a superset of C control flow semantics. Pick different names in vacietis.c and you have an AST.

I don't know what the title was when you commented, but what I see is "Compiler using Lisp’s macro system for metaprogramming C-like languages". That doesn't imply transpiling from Common Lisp (though it does impli transpiling), and it seems like a very pithy description of what C-Mera actually does (which is also what you said in your first sentence).

The nice thing about this is that you get the full power of Lisp macros, which is where the metaprogramming comes in, but it still resembles C well enough.

Of course, the Lisp compiler can't actually do C type checking in this system, so you still need to map C compiler errors and warnings back to the source -- that may well turn out to be a pain (and almost certainly does).

EDIT: I'm curious: why the downvote?

TLDR -- there is nothing so special about Lisp (in terms of metaprogramming) other than a syntax that is easy to parse. All languages have tools to manipulate code at the source code level, but people rarely use them. These guys did.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact