
Pixie – A small, fast, native Lisp - throwaway7645
http://pixielang.org/
======
_halgari
I'm the original author of pixie, and yeah, I'm a bit surprised to see this
hit HN today.

It should be mentioned that I put about a year of work into this language, and
then moved on about a year or so ago. One of the biggest reasons for my doing
so is that I accomplished what I was looking for: a fast lisp that favored
immutability and was built on the RPython toolchain (same as PyPy). But in the
end the lack of supporting libraries and ecosystem became a battle I no longer
wanted to fight.

Another goal I had was to see how far I could push immutability into the JIT.
I learned a lot along the way, but it turns out that the RPython JITs aren't
really that happy with VMs that are 99.99% pure. At one point I had a almost
100% immutable VM running for Pixie...as in each instruction executed created
a new instance of the VM. It worked, but the JIT generated by RPython wasn't
exactly happy with that execution model. There was so much noise in the
maintenance of the immutable structures that the JIT couldn't figure out how
to remove them all, and even when it could the JIT pauses were too high.

So anyways, after pouring 4 hours a day of my spare time into Pixie for a full
year, I needed to move on.

Some other developers have commit rights and have pushed it along a bit, but I
think it's somewhat a language looking for usecase.

And these days ClojureScript on Node.js could probably be made to handle most
peoples needs.

~~~
samth
As a somewhat different data point, we've been developing pycket, an
implementation of Racket on top of rpython, for the past 3 years, and while it
faces many of the same challenges, we've been very happy with the results. The
JIT can remove almost all of the intermediate data structures caused by the
functional nature of the language, and we support tail calls and first class
continuations. Overall, pycket is almost as fast as chez scheme on average,
and faster than every other scheme system we've compared with.

~~~
shakna
Interested in the speed you're seeing.

I use Gambit-C scheme for most of my scheme needs because of the speed, but it
seriously lacks in libraries, that Racket has in abundance.

~~~
baldfat
[https://github.com/pycket/pycket-bench](https://github.com/pycket/pycket-
bench)

~~~
sabauma
Much of the data in that repo is old/outdated. This
([https://github.com/pycket/pycket-
bench/blob/master/output/20...](https://github.com/pycket/pycket-
bench/blob/master/output/20150610_fbc4c2d_CrossBenchmarks-norm.pdf)) figure
was the one used in the paper, which includes Gambit.

------
copx
> _And it 's small. Currently the interpreter, JIT, GC, and stdlib clock in at
> about 10.3MB once compiled down to an executable._

Oh how the definition of "small" has changed. I actually would like to know
how they managed to make something like this so _big_.

To compare, LuaJIT is about 400 KB, and that includes the Lua standard
library, a JIT almost certainly more advanced than Pixie's current one, an
incremental GC, and a C FFI.

Neither compilers (well, except C++ ones), nor stuff you usually find in
standard libraries, nor a GC should require much code to implement, relatively
speaking (e.g. compared to a WYSIWYG word processor). These things are
_usually_ small. The compilers for almost every language were < 1 MB in size
for the longest time.

I am not saying that Pixie being 10 MB in size is a problem. We have a lot
more bandwidth and disk space nowadays, 10 MB is nothing. My point is that a
"JIT, GC, and stdlib" package weighing this much cannot claim to be "small"
for what it does.

~~~
stcredzero
_To compare, LuaJIT is about 400 KB, and that includes the Lua standard
library, a JIT almost certainly more advanced than Pixie 's current one, an
incremental GC, and a C FFI._

Back in the day, Smalltalk was criticized as bloated because one ended up with
stripped binary+image of about 2MB. Someone around that time got
SqueakVM+image down to just under 400k. There were specialized Smalltalks in
company R&D that got their image down to 45k.

~~~
beamatronic
Don't forget, even 2 MB was still a somewhat inconvenient size when floppy
disks were only at 1.44 MB capacity.

~~~
stcredzero
Floppy disks were well on their way out, and people were still complaining
about a 2 MB footprint.

~~~
timClicks
I thought Smalltalk's hayday was early-mid 80s. That would imply 5.25"
floppies would still be in use and 3.5" floppies overtaking them circa 1988.

~~~
stcredzero
_I thought Smalltalk 's hayday was early-mid 80s._

My involvement with Smalltalk started in the 90's. Then it still had high
penetration in the Fortune 500, and it was a niche language for complex
financial programs. It was also used in Energy.

------
xaduha
[http://www.red-lang.org](http://www.red-lang.org) for those who doesn't
necessarily need Lisp, but appreciates homoiconicity and other nice things.

> This executable is a full blown native interpreter with a JIT, GC, etc.

> A small, fast, native lisp

I have issue with the use of the word 'native' here. For me 'native' mostly
means AOT compiling.

~~~
chrisseaton
They said that the _interpreter_ was native. It is - it's AOT compiled. They
didn't say that your _program_ was natively compiled. Your program is
interpreted - by the interpreter - the native intepreter - and then JIT
compiled by meta-tracing the interpreter.

Their terminology is totally consistent with how the field uses these terms.

This technology operates at multiple levels of meta-implementation, so it is
easy to get confused what is tracing what, and what is implemented using what
at a given time.

~~~
xaduha
It's useless information then, any real executable is native. In that sense
you can say Perl is native (Python, Ruby, JavaScript/nodejs).

~~~
chrisseaton
Well yes any real executable is native. The perl executable is indeed a native
program. It's a native interpreter for Perl.

But there are also interpreted interpreters aren't there? Which aren't real
executables, and aren't native. It would be possible to write an interpreted
interpreter for Perl, maybe in a language like Ruby. Jython is a real example
of an interpreted interpreter (if you ignore that the JVM has a JIT, but it
isn't AOT, which you've said that you think is an important criteria).

And so it isn't useless information.

The distinction is particularly relevant here because the RPython technology
they are using to build their interpreter means they can either interpret
their interpreter using the Python interpreter, or they can make a native
interpreter by compiling their interpreter to native AOT. That's probably why
they used that particular wording.

So, again, not only is their terminology consistent with the rest of the
industry, they are also making a specific and interesting point here, and it
isn't useless information.

~~~
xaduha
They put that in a headline - A small, fast, native Lisp. It's
missleading/clickbaity.

~~~
kbenson
I think it's meant as in comparison to Scala and the like, which run on the
JVM, and thus aren't "native" in the same way (right? or am I misinformed
about Scala?). There's a lot of languages that sit on top of the JVM at this
point, so they might have seen it as a distinguishing characteristic.

~~~
jibalt
> thus aren't "native" in the same way

Sorry, no, this is nonsense. Scala source code is compiled to Java bytecode,
which is interpreted and JITed by the JVM. Pixie source code is compiled to
the Pixie bytecode which is interpreted and JITed by the Pixie VM. chrisseaton
claims that "native" refers to the interpreter, but this is rubbish ... that's
not what the industry means by "native".

------
jlarocco
Any language claiming to be a fast Lisp should be required to show benchmark
comparisons against SBCL and a few commercial Common Lisps.

A native compiler for Clojure would be an interesting project, but completely
new languages competing on speed or size are going to have a really tough time
beating the existing Common Lisp implementations, not to mention the CL
library ecosystem.

Not to say the CL library world is very big compared to Python or Javascript,
but it has most of the important bases covered, and it's certainly bigger than
a brand new language like Pixie.

------
branchly2
Alas, this ambitious project appears to be not currently under active
development.

My largely uninformed armchair opinion as to why, is that the author is very
performance-driven, and in the end it's very difficult to beat the JVM
performance-wise. Lesson: if you want high-perf Clojure, you already have it
on the JVM.

Personally, I think there's room for a simple small native Clojure
implementation where performance is not top-priority. Small footprint, quick
startup, access to native C libs. Still holding out hope for that one.

~~~
yellowapple
I'm generally in demand of small, quick startup, easy-to-embed languages that
aren't Lua (I can't stand Lua; I feel like every time I've had to deal with
Lua code was far more painful than reasonably possible). There's mruby and
maybe a couple others. I was hoping Pixie might be another contender in that
space, but it doesn't seem to be after all (or if it is, it's very poorly
documented to that effect).

I've been tinkering with some ideas for another language in that space (the
Crafting Interpreters book is looking to be pretty helpful in getting me
there:
[http://www.craftinginterpreters.com/](http://www.craftinginterpreters.com/)),
but it'd be really nice if there were enough options for me to not feel the
need to create my own.

~~~
networked
A list that I maintain might help you find one:
[https://github.com/dbohdan/embedded-scripting-
languages](https://github.com/dbohdan/embedded-scripting-languages).
Personally, I am a fan of Jim Tcl ([http://jim.tcl.tk](http://jim.tcl.tk)),
especially for when you need a small interpreter that knows how to interact
with its Unixy environment (i.e., the file system, processes, sockets and
UTF-8 text data) out of the box.

~~~
yellowapple
Jim Tcl actually looks about damn perfect. I'll definitely have to give that
one a whirl.

------
mschaef
So this is great, but note that development has seemingly fallen off to close
to zero over the last year:

[https://github.com/pixie-
lang/pixie/graphs/contributors](https://github.com/pixie-
lang/pixie/graphs/contributors)

------
Sir_Cmpwn
Interestingly, this is (some of) the output of running the makefile:
[https://sr.ht/zMyE.png](https://sr.ht/zMyE.png)

~~~
sabauma
The RPython translator generates views of the Mandelbrot set by default,
presumably to give you something to look at during the long wait.

~~~
s_kilk
That's pretty cool.

I kinda wish the likes of NPM would output cool skeletons and heavy-metal
vampire chicks while I wait ten minutes for it to install stuff.

------
sigjuice
I find it very hard to work with a mixture of brackets and parentheses. Does
anyone else prefer an all-parentheses syntax for reading and typing?

~~~
hkjgkjy
Well, in most cases `[]` is used just like you'd use a quoted list `'()`.
Clojure vectors are nice for the performance they give.

Having maps with `{}` syntax is also nice if you ask me. Makes reading a
little bit faster.

~~~
junke
> Clojure vectors are nice for the performance they give.

What difference does it make when representing code?

~~~
hkjgkjy
What I meant was that [] when glancing just means the same as '().

I was thinking about it over the day, and realized maybe having special syntax
actually is confusing - maybe "normal" lisp with only parens is better. Thanks
for the eye-opener.

~~~
junke
Thanks for your comment.

The idea behind the Lisp reader approach is that custom syntax is used for
reading/printing objects of different types, not to demarcate syntactic
elements. For example, double quotes are for strings, #P"" will read
pathnames, and so on. In Common Lisp, some characters like [ and { are
reserved for the user, meaning that no conforming implementation defines a
custom syntax based on those characters. And you can define [a b c] to mean
(vector a b c), which will build a vector, when executed, to hold the current
values of a, b and c. The existing #(a b c) vector syntax is a literal vector
that contains symbols.

From this point of view, for Clojure, I don't think it is a bad idea to have a
short syntax for vectors, set and map literals. But then, those objects are
used when representing code, not because it has a real added value but because
the visible syntax is a little bit nicer (maybe, maybe not). That IMO
complexifies tools that work with code and does not really fulfill a practical
purpose. If that was for practical reasons, I think bindings would had been
better defined as maps: as far as I know, they seem to fit more naturally than
vectors for this task. For example:

    
    
        (let {a 20 b 10} ...)
    

Somehow when compiling or interpreting the code, you could do "(get symbol
env)" where "env" is the surrounding lexical environment associated with the
let, which would be computed partly from a parent lexical environment and {a
20 b 10}. The environment and the map could be of the same type and be easy to
combine. But there is no such consideration, and it is actually a good thing
that there is no link between how the code is represented and how it is
interpreted (also, there can be different interpretations).

Basically, I find Clojure a little bit confused about its use of external data
representation for code. I would have preferred a simpler syntax for code and
keeping those notations for data, where they are self-describing without
additional context.

~~~
gw
Hash maps wouldn't work for bindings because they don't guarantee order,
whereas the order of the bindings is very important.

~~~
junke
You are right, I forgot about that (the example was not great anyway). Thanks

------
spraak
> _If you like Clojure, but are unhappy with the start-up time, or if you want
> something outside of the JVM ecosystem, then Pixie may be for you_

YES! :D

~~~
josteink
In that case... what about just regular plain old Common Lisp?

~~~
DannyB2
I'm not intending to criticize Common Lisp, which I loved back in the early
90's, but there are things to like about Clojure over Common Lisp.

~~~
na85
Such as?

~~~
DannyB2
A big one is persistent data structures. Where 'persistent' may not mean what
you think it means.

In Clojure it is impossible to surgically modify a data structure. That is,
you can't do something like:

(SETF (CAR (CDR x)) 'foo)

which would alter a data structure.

You can modify a data structure, but it returns a new data structure, yet the
old one remains if it is not GC'able.

All of the common data structures have this property. Sequences (eg, lists),
arrays, maps and sets. If you change the 50 thousandth element of an array,
this returns a new array. The old array is unaffected. Yet it gives the
performance you expect of an array. (Meaning no apparent cost of copying.)

If you're writing a search procedure, it is trivial to transform one
chessboard into a different chessboard, but without concern about the cost of
copying (close to zero), or having altered the original value (you haven't).
Other variables that have a pointer to that first chessboard don't see any
changes.

There are other things such as a great story about concurrency.

Hope that helps.

~~~
inconclusive
How is the cost of copying close to zero or not apparent?

~~~
rads
The data structures in Clojure are built on shallow 32-way trees. When you
"change" a map or a vector, the algorithm only copies from the root node down
to the parent of the leaf you're changing. That's log32(N) copied nodes in a
tree of N nodes total.

------
lvh
As someone who loves both PyPy and Clojure, I wish Pixie was right up my
alley. But I don't understand why the choice was made to be ... sort-of like
Clojure, but not all the way, as opposed to just-another-Clojure-
implementation. Why can't I have a .cljc file that runs on Pixie?

(Not voluntelling halgari to do things! I would just like tn understand.)

~~~
mschaef
> sort-of like Clojure, but not all the way,

I tend to agree... but doesn't 'all the way Clojure' imply at least some
exposure of an underlying runtime, be it either Java (Clojure) or JavaScript
(ClojureScript)?

I'm admittedly a bit behind the curve wrt the latest developments in the
Clojure along these lines.

~~~
lvh
Yes! Being able to call PyPy things natively would be a godsend; PyPy is a
great runtime and already has a good Python implementation. It has
significantly fewer compatibility issues than, say, Jython.

~~~
omginternets
I'm taking a shot in the dark, but have you looked at the Hy language? I
imagine it must be relatively straightforward to port this over to RPython.

~~~
lvh
Hy is also great, but it is even further away from Clojure than Pixie is.

------
mikaelj
As always, for all these small languages (MicroPython comes to mind, but even
"real-world" languages such as Lua) - unless they grow a debugger, they'll
always be silly toy languages nobody can use for serious work.

I'd settle for a gdb backend, really. But printf-debugging is unacceptable.

------
JohnStrange
I like it, will definitely take a look at it. The name is also good, it
reminds of its similar cousin Pico lisp and the website is nice. Now we need
libraries, many of them, and better docs. Good work!

------
abc_lisper
What's the state of development of this? Did somebody use it with Raspberry
pi? I want to use this over Clojure(only on pi) because I have read Clojure is
slow on Raspberry pi(even 3, not sure how true that is).

Also, how fast is this? How does it compare to other lisps(or schemes) in
terms of speed? Can someone port the benchmarks to
[https://benchmarksgame.alioth.debian.org](https://benchmarksgame.alioth.debian.org).

~~~
throwaway7645
I posted the link, but am not one of the developers. This got a little
attention a couple of years ago, and I thought to look it up today. It has a
nice website now at least. I really like the idea behind it (small install,
lightweight, fast), but don't know how effective it will be without a larger
community. I can honestly do without a lot of libraries as long as there are
decent built-ins.

I really wish Pixie just had a REPL and compiled to native code via LLVM so I
could give coworkers a .exe.

Edit:

Just looked at the github page. It doesn't look very active to me although I
wouldn't say dead. The only small lisps I know of in development are Picolisp
and Newlisp.

~~~
jacobush
PicoLisp is very small and extremely powerful at the same time. It has a built
in Prolog...

~~~
throwaway7645
Do you use it though? I really like the concepts, but the speed could be
better (I know Alexander talked about this), and docs could be more beginner
friendly. It's a small expert community.

~~~
jacobush
I have used it in commercial embedded hardware shipping tens of thousands of
units, in the USD 1000 range. Speed is excellent for the most part - where
it's not it's trivial to call out to C libraries. Agree about docs but what
would really help at this point would be more QA on Stackoverflow. And a good
Windows port.

~~~
throwaway7645
Agreed on the Windows thing. I tend to stay away from Cygwin. I like the new
Picolisp site btw. The code on Rosetta code is terse, and uses slightly
different idioms than I'm used to.

~~~
jacobush
Come to think of it, some docker and other fancy container magic could also
help adoption. And other such things people use nowadays. Maybe a nodejs
integration? Stuff like that.

~~~
throwaway7645
My dream would be to just use something like Picolisp on Windows without the
workarounds. Maybe that can be done via the new Bash on Windows10? Idk.

~~~
jacobush
Must be possible with the new Ubuntu layer whatever its called. Edit:
[https://blogs.windows.com/buildingapps/2016/03/30/run-
bash-o...](https://blogs.windows.com/buildingapps/2016/03/30/run-bash-on-
ubuntu-on-windows/)

They say WSL apps can not interact directly with Windows apps. So yes and no
to your question. I would say it would be similar to running in a virtual
machine but more integrated.

------
eximius
Very neat but documentation is _very sparse_ , from what I can tell.

------
eschaton
Why does the web site refer to "Lisp interpreters?" That phrasing perpetuates
mistaken assumptions about Lisp-family languages, both about their
implementation and their performance.

------
jyriand
Do I understand correctly that pixie is using clojure's lisp dialect and the
only difference is that it is not running on JVM?

------
endergen
This is going to get confused with Pixi, the more and more popular 2D
rendering engine for Javascript:
[http://www.pixijs.com/](http://www.pixijs.com/)

~~~
iLemming
So what? Everything has a suffix in js-land.

------
chubot
In my experience, whether code like summing integers takes 6 instructions or
600 doesn't influence the speed of 95% of code and 95% of systems.

If you could magically port real Python or ruby or JavaScript code to pixie,
but keep the same algorithms and architecture, I doubt it would change much.

Slowness is more influenced by things like data structure layout, allocation
patterns, serial vs parallel I/O, context switches, and just plain not
understanding your code once it reaches a certain size.

There seems to be a fetish for JIT compilation in a lot of new language
designs and it confuses me. Julia is probably the language making the best and
most appropriate use of it. It actually has good data structures and types
which complement it.

~~~
dinkumthinkum
What do you mean in your experience? Summing integers can take two orders of
magnitude more time and it doesn't matter? I think it goes without saying in
discussions of performance, we narrow our focus to performance sensitive
applications and not CRUD database web frontends. I can tell you in the vast
majority of performance sensitive numerical applications, that would matter.
It would have to be massively disk bound for it to be overshadowed completely
by I/O. Just saying. :)

~~~
chubot
I don't follow. Why aren't database web front ends performance sensitive? They
seem like one of the slowest things out there, and some of the most widely
used. My comments here on Steve Souders' realization is basically what I'm
talking about:
[https://news.ycombinator.com/item?id=13346635](https://news.ycombinator.com/item?id=13346635)

If you want to make systems fast, you work on the bottleneck. I'm saying that
people think too often that summing integers is the bottleneck, when it
plainly isn't.

Although I have worked in the domain of numerical applications, thus my nod to
Julia. Anybody who works in that domain isn't going to be using something like
Pixie; it's too impoverished in terms of types and data representation.

