
Mezzano – An operating system written in Common Lisp - arm
https://github.com/froggey/Mezzano
======
DonaldFisk
Very impressive.

Since the demise Lisp machines, there have been several attempts at developing
a Lisp-based operating system which didn't deliver anything: LispOS, Tunes,
and Loper; and one successful attempt at getting Lisp to run on the bare
metal: Movitz.

Lisp was its own operating system on the original Lisp machines developed at
MIT, and this evolved into Genera at Symbolics. Is this the case here?

I notice that Mezzano has files. Files are inherited from operating systems
running non image-based languages. They're alien to Smalltalk, where
everything is just stored in the image (or world, in Lisp parlance). This
could also be done in Lisp. Text files could be replaced by long strings
contained in the world. Hypertext, provided tags are balanced, could be stored
as lists. When the system is shut down, and periodically beforehand, the world
is written to disk.

You can go further, by using a single address space. Particularly large blocks
of text which don't fit into RAM, and so have to be stored on disk, can be
addressed by treating the disk as an extension of RAM. You can even go further
on 64 bit machines by treating the entire contents of the internet as an
extension of RAM.

I once wrote a small bare-metal Lisp interpreter in x86 assembler, which ran
off a 1.4Mb floppy on an old laptop. The hardest part was writing the floppy
disk driver, which has to handle frequent hardware errors. USB drivers are
many times harder and I didn't attempt that. If the Mezzano developer(s)
succeeded, I'm even more impressed.

~~~
spaceheeder
I think files have proven themselves to be a useful abstraction in enough use
cases across enough device form-factors to be worth backporting into whatever
your idea of an ideal lisp machine might be.

You could, with a non-trivial amount of effort, replicate the file-like
conveniences of global tagging and sorting and organizing of all the objects
in your image. You could, also with a non-trivial amount of effort, work out
the schemes for permissions, etc. so that objects with all those file-like
conveniences can be shared, like files, on multi-user systems or between
machines or over networks.

It's not clear to me that the above would offer any tangible benefit over
files. So if you're going to put a non-trivial amount of effort into a lisp
machine, why not just teach it what a file is?

~~~
DonaldFisk
> I think files have proven themselves

You will need to store data which doesn't fit into RAM in secondary storage.
But that doesn't mean you need a file system or even files.

> to be a useful abstraction

Files are a necessary evil in non-image based systems because you need to
store data somewhere when the programs using them aren't running. As the
different objects they contain, such as plain text, hypertext, photographs,
sound recordings, executables, etc. have nothing in common, they seem an
unnecessary abstraction. They require that programs which use them
parse/serialize their contents. This is unnecessary if the contents are
already in memory, already in the format the program needs.

> replicate the file-like conveniences of global tagging and sorting and
> organizing of all the objects in your image

Why remember a file name and where it is in the directory tree when you could
use a search engine to search for it based on content? Or simply chain through
objects, going to the field you want and following the link? Programs, of
course, will just directly link to the object.

> work out the schemes for permissions

I'd go for capabilities, rather than access control.

> why not just teach it what a file is?

Building a file system is a major undertaking. If it can be avoided, and to
the extent it can be avoided, it should be. You would only need to know about
files when you interact with systems which are based around files.

~~~
TeMPOraL
I think files are pretty cool because they're _loosely coupled_ with programs.
I.e. you can open and edit text or audio files in whatever you like, and even
highly-specific binary formats like .doc or .psd have various levels of
support in 3rd-party tools. This serves a very useful purpose of ensuring your
data is truly yours, and will outlive the program in which it was created.
This is a place where IMO we're taking a _huge_ step backwards now, with
increasing amount of work being done in the cloud and mobile ecosystems - both
shed the concept of files for some amorphous database entries somewhere, thus
taking away your control over your data.

Can you preserve this flexibility/loose-coupling feature in image-based
storage? I don't know. I'd be interested to learn if it can be done so.

------
spaceheeder
One of my favorite things to learn when I come across any experimental OS is:
"why?" Redox wants to do microkernel architecture The Right Way, and wants to
use a guaranteed memory-and-type safe language (Rust). ReactOS wants to create
a drop-in replacement for Windows that not only supports older Windows
programs, but also Windows-compatible device drivers. MenuetOS wants to build
something approximating the OS experience most users are used to, but on bare
metal in assembly.

So: why Mezzano?

~~~
informatimago
Basically, the point of lisp OSes, is that they allow you to seamlessly write
code at all layers, from down into the darkest bowels of the system, thru and
up to the highest level application scripting, all in a single integrated
language and libraries and frameworks.

You don't have to reboot a machine just because you make a patch to the
kernel, and at the same time, you can patch the system using the same high
level development tools as you would use to develop an application.

The thing is that there's what we call an impedence mismatch between a unix
kernel and applications running on it, in that the data types processed by
applications which are of higher level, don't match the data types processed
by the unix kernels. For example, a Ruby Integer is actually a bignum, and you
cannot pass a bignum to the unix kernel: you have to provide a bit field with
32-bit or 64-bit. This is work, and this is pain. It is already pain when you
write your applications in C or C++ where assumedly you already have bit field
types, because your application could be compiled to run on kernels and
processors using different word width! Imagine the pain it is when you write
your applications in Common Lisp with very high level data types (ratios,
closures, pathnames!), and when you have to map those data objects onto the
lame bits accepted by unix syscalls. Consider all the literature written about
logging, the byte vector logging provided by unix syslog vs. high level object
logging provided by higher level libraries or other systems.

And of course, this is not limited to the interface between applications and
systems, but goes beyond to the interface between applications. When the
system provides a pipe abstraction where all you can pass between applications
are streams of byte, this leads to a lot of suffering. You have to serialize
and deserialize your data, you have to consider formats (a java float doesn't
have the same syntax as a Common Lisp float!), encoding (utf-8? iso-8859-1,
-15?). Have you ever been advised to never parse the output of /bin/ls? For
good reasons!

Instead, if your system is written in Lisp, then you can directly pass lisp
objects from one lisp application to another lisp application, and there's no
need to serialize/deserialize, to parse or otherwise mangle the data: you just
have lisp objects and you can use them directly. You can pass closures (which
enclose the lisp object data along with the lisp functions needed to process
them).

~~~
catnaroek
> Instead, if your system is written in Lisp, then you can directly pass lisp
> objects from one lisp application to another lisp application, and there's
> no need to serialize/deserialize, to parse or otherwise mangle the data: you
> just have lisp objects and you can use them directly. You can pass closures
> (which enclose the lisp object data along with the lisp functions needed to
> process them).

So, if I understand correctly, the memory block in which a particular object
is stored doesn't belong a priori to this or that process, but rather you can
simply hand down objects to other processes? This is very cool! I've been
thinking for quite a while that the very idea of memory protection is just a
lame workaround to deal with the fact C is memory-unsafe.

~~~
parenthephobia
From what I can tell looking at the code, Mezzano doesn't have processes, just
threads. There's no distinction between the kernel and user-space.

Whilst convenient, it isn't an unqualified good. Bugs in any part of the
system have the potential to catastrophically break everything. It's also
terrible for security. |Without the notion of a distinct kernel - or "unsafe"
code that only privileged users can compile or load - there's nothing to stop
user code reading or writing directly to/from IO ports or arbitrary memory
locations.

Being able to pass closures isn't something that's particular to Lisp, nor
does it require a lack of memory protection. It just requires that the calling
convention can denote a closure as such, and can arrange for it to be invoked
in the right security context - i.e. the one in which it was created.

Another issue with having a single global environment backing everything is
that you can't easily experiment with changing built-in Lisp functionality
without immediately crashing the system.

This sort of thing is probably perfectly fine for embedded systems, and
"micro-service" VMs, though. The benefits may well outweigh the risks then.
And of course it's great as a starting point for experimentation.

~~~
joshmarlow
> It's also terrible for security.

The security angle is interesting.

The thing about independent processes is that we're relying on the OS to
provide a wall around each process to limit it's impact _by default_. Maybe a
better approach would to be include something like chroot/containerization for
execution of closures _within the language_? Ie, if program 'A' receives data
from program 'B', it can evaluate that data in a sandbox with limited
permissions (IO limitations like no networking, or CPU resource limitations
and time limits) and receive exceptions if the closure tries to exceed it's
permissions. That could fit in pretty cleanly with Common Lisp's condition
system.

A con of this though: you could argue that it _really_ breaks encapsulation;
it's bad enough when different parts of a large program become too closely
coupled - allowing different applications to depend on the internal state of
each other could be a nightmare.

~~~
nickpsecurity
Relevant links here:

[https://news.ycombinator.com/item?id=12362583](https://news.ycombinator.com/item?id=12362583)

~~~
joshmarlow
I've really enjoyed your link-sharing lately. Thanks for spreading knowledge
:)

------
erikj
Wow, this is pretty impressive. The last time I checked its progress, it was
very basic. Now it even has a working GUI with transparency! I should find
some time to tinker with it.

------
johnhattan
I hope there'll be a driver for the Symbolics keyboard. I cannot be productive
without a "meta", "super", and "hyper" key.

------
rcarmo
Neat. I rather like the idea, although these days any OS that doesn't target
embedded systems and doesn't sport a web browser is instantly filed as a tech
demo...

------
zeveb
What'll be really awesome is once it's possible to run SBCL on Mezzano (or
would it be Mezzano in SBCL?), to take advantage of all the optimisations SBCL
offers.

------
qwertyuiop924
This is a fun toy.

As our forefathers did program upon the metal in Zetalisp, now thou shalt
program upon the metal in Common Lisp.

:D

------
blue1
I see instructions to install it in a VM, but is it possible to run it on bare
metal?

~~~
qwertyuiop924
I assume so: The only real issue would be BIOS compatability, so it might take
some work, but it should be possible.

~~~
vertex-four
Well, the main issue would be it requires a specific network card to talk to
its file server.

~~~
qwertyuiop924
Ah, missed that. Yep.

I think USB, etc. are standard enough, so that should be it.

With Intel motherboard, and Intel integrated networking, it could probably run
in my machine.

Wait. Graphics. %$&#!.

~~~
khedoros
Depends on how graphics are implemented, right? Even a modern graphics card
has some kind of support for older standards like plain VGA and VESA video
modes, it seems like.

~~~
qwertyuiop924
Yeah, but I don't even know if Mezzano queries for a GPU or graphics card. It
might just decide to use onboard no matter what.

------
mikecsh
Somewhat off-topic but can anyone point me to any resources that explain the
hardware architecture differences between x86 CPUS and the old LISP-machines?
Was there hardware level support for s-expression evaluation?

~~~
varjag
They were microcode machines with certain optimizations for handling Lisp data
structures. The memory had also extra tag bits per every word for the benefit
of runtime type dispatch and garbage collection.

------
dpc_pw
Wouldn't it be easier to port emacs to common lisp?

Hmm... Actually, probably not.

~~~
lispm
Depends what you call 'port' and 'emacs'.

Common Lisp already has some native variants of Emacs. Though not GNU Emacs.

The first one was called Hemlock and was developed in the early 80s.

[https://en.wikipedia.org/wiki/Hemlock_(editor)](https://en.wikipedia.org/wiki/Hemlock_\(editor\))

------
sigzero
That's pretty nifty. I will have to fire up a VM and try it out.

