
Symbolics Genera - The Best Software Environment Available - udzinari
http://lispm.dyndns.org/genera-concepts/genera.html
======
ScottBurson
I have a Symbolics XL1200 that still works (at least, it worked the last time
I turned it on, which was a few years ago). I'd be happy to demo it at YC
sometime.

It was a fun environment to work in. Although the hardware was very slow by
modern standards, the environment was designed to maximize hacker
productivity. For example, you didn't have to run a program under the debugger
to debug it -- the debugger was always there. Any program that hit an error
would drop you into the debugger. One keystroke would then take you to the
source for the current function. You could change that function, recompile it,
and restart from the point where the function was called.

The environment was completely open; there was no access control of any kind.
There was also only a single address space. Did that make it crash-prone?
Remarkably, no. The hardware tagging and bounds checking kept programs from
stepping on objects they didn't own. Of course such a design would never
survive outside of a research lab, but it made it amazingly easy to hack
almost anything in the system.

~~~
daeken
I don't know that I'd agree that it wouldn't survive outside of research. In
recent years, there's been a huge push for pure-managed operating systems,
where everything runs in one address space and depends on the compiler to
ensure security and reliability. While no such systems are in practice (to my
knowledge), the potential is incredible. We'll see how it pans out in the next
few years.

(Full disclosure: I started such an OS, Renraku, so I have a bit of a vested
interest in the success of such ideas)

~~~
joe_bleau
What about IBM's OS/400? <http://users.snip.net/~gbooker/AS400/ibm.htm>

~~~
hga
Indeed, OS/400 is just wonderfully advanced under the hood. Get one of the
cheap used copies of this book for a lot more: [http://www.amazon.com/Inside-
AS-400-Frank-Soltis/dp/18824196...](http://www.amazon.com/Inside-AS-400-Frank-
Soltis/dp/1882419669/)

------
julian37
Wikipedia has more information for people (like me) who haven't heard of
Genera before: <http://en.wikipedia.org/wiki/Genera_(operating_system)>

------
sedachv
Lisp Machines are something that you think is really cool when you first learn
about them, then you come to the realization that pining for them is a waste
of time.

I've had a flash of inspiration recently and have been thinking about Lisp
Machines a lot in the past three weeks.

But first, a digression. There's an important lesson to be learned about why
Symbolics failed. I think Richard Gabriel came to the completely wrong
conclusion with "Worse is Better"
(<http://www.dreamsongs.com/WorseIsBetter.html>). There are two reasons why:

1\. Out of all the LispM-era Lisp hackers, only RMS understood the value of
what's now known as Free Software. (If you haven't read it yet, read Steven
Levy's Hackers - it describes the MIT/LMI/Symbolics split and how RMS came to
start FSF and GNU).

2\. Portability is really important.

The key lesson to draw from Unix isn't that "Worse is Better," it's that
survivable software is Free and portable. Free because getting software to
someone's harddrive is 80% of success, and portable because you don't know
where people will want to use your software (there are some really weird
places).

Symbolics was neither. If Genera had been Free Software, it would by
definition still be around today. If Genera had been portable, it's likely
Symbolics would never have gone out of business (the Alpha virtual machine
would have been done sooner, with less resources, and for more systems).

Being released as Free Software today wouldn't help. Genera's predecessor, MIT
CADR, was made available under an MIT-style license in 2004
(<http://www.heeltoe.com/retro/mit/mit_cadr_lmss.html>). There's a VM emulator
which runs the code. The whole system is pretty useless.

 __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __
__ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __

Now on to the inspiration part:

It's possible to make a very high-performance, portable Lisp operating system
on modern hardware. This has been a possibility ever since the Pentium came
out. The main bottleneck to conventional Lisp runtime performance is the way
operating systems manage memory allocation and virtual memory.

A type-safe runtime that has control over memory layout, virtual memory, and
is aware of DMA can provide extremely high throughput for allocation and GC
(this has been shown by Azure's Linux patches for their JVM), true zero-copy
I/O, almost optimal levels of fragmentation, and excellent locality
properties. If you go single address space (and there's no reason not to) and
move paging into software (object faulting and specialized array access),
you've also eliminated TLB misses.

Throw in the fact that it now becomes trivial to do exokernel-type stuff like
for example caching pre-formatted IP packets, and it should be possible to
build network servers that have throughput many times that of anything that
kernel/user-space split OSes like Linux or FreeBSD are capable of for dynamic
content (ie - not just issuing DMA requests from one device to another).

The only problem is device drivers. Lisp doesn't make writing device drivers
any more fun, or reduce the number of devices you have to support.

What to do?

The reason I've been thinking about this is that I came across this:
<http://www.cliki.net/Zeta-C>

I've heard of Zeta-C multiple times before, but for some reason this time I
made the connection - "why not use Zeta-C to compile an OS kernel?"

I explored the idea further, and it seems to me that it wouldn't be an
unreasonable amount of work to take the NetBSD device subsystem and have it
running on top of a Lisp runtime with the necessary emulation of those parts
of the NetBSD kernel that the drivers depend on. If you don't know, NetBSD's
device drivers are modular - they're written on top of bus abstraction layers,
which are written on top of other abstraction layers (for example, memory-
mapped vs port I/O is abstracted). So the actual system twiddling bits can be
neatly encapsulated (which isn't necessarily true for Linux drivers, for
example).

I'm aware of Movitz (<http://common-lisp.net/project/movitz/>) and LoperOS
(<http://www.loper-os.org/>). Movitz makes the mistake of trying not to be
portable, but there's useful things there. I haven't spoken to Slava about
this yet so I don't know what's going on with LoperOS. I am also aware of
TUNES, and think it was an interesting waste of time.

The main thing is to get Zeta-C to work on Common Lisp. Then it's to build a
new portable, boot-strappable runtime (I think the Portable Standard Lisp
approach of having a SYSLISP layered on top of VOPs is the right way to go for
this), and either build a compiler targeting that runtime, or adapt the IR-
generating parts of one of SBCL, CMUCL or Clozure. Further bootstrapping can
be done with SWANK and X11 once a basic networking stack is in place. I think
such a system would be quite fun to hack on.

If you've gotten this far, let me know what you think about this idea. I also
have some preliminary thoughts about how this can be worked into the base of a
new high-performance/scalability transactional database startup, if you want
to hear about that email me: vsedach@gmail.com

~~~
lispm
Steven Levy basically repeats Stallman propaganda. Others have different
memories about that time.

Also not only Stallman promoted free software in that time. For example the
Common Lisp community had extreme benefit from what was done at CMU.
CMUCL/Spice Lisp was released as public domain software and has been used
widely in derived implementations, like SBCL or even commercial software.
There were several other prominent Lisp hackers from that time which promote
free software.

Stallman did a lot, but please make him a saint - when people like Scott
Fahlman had much more impact in the Lisp community.

~~~
tangus
Doesn't that reinforce his point? CMUCL & SBCL were released as free software
and are still around. Genera isn't.

~~~
lispm
I haven't said anything against his point. It is just that claiming that
Stallman was the only one 'getting free software' is bullshit. He did a lot,
but in the Lisp community there were many others which promote(d) free
software.

Also Stallman did promote Free software, but his promotion of Lisp was mostly
limited to Emacs Lisp. There was an attempt with Guile, but that was mostly
it. He had worked with the Lisp Machine, but once he was out, that topic was
finished.

Anyway, I find a few points are right and several others not. That Symbolics
would have not gone out of business if Genera would have been free software
and portable is speculation. It is neither necessary nor sufficient. For
example Franz is still around, though Allegro CL is not free software. There
are lots of free software / portable Lisp implementations which are as dead as
Genera, or more than dead. I agree that it might help, but not more.

I agree with the following: portability and 'free' can help with survival.

I don't think adding C to Lisp is of much help. The mismatch is huge. There
are two options: 1) repeating the design of the past (with the same problems
and maybe slight improved. There has been lots of talk and little action. It's
difficult because we don't have the hackers with knowledge and time for that.
2) doing a new design which gets rid of the limitations of past designs
(single user, single image, no security, ...). That's even more difficult.

~~~
sedachv
"That Symbolics would have not gone out of business if Genera would have been
free software and portable is speculation."

Two different things. I was claiming that Genera would still be around if it
was Free Software - this is by virtue of definition. Symbolics being in and
out of business has nothing to do with it.

A totally different thing is that I think Symbolics stood a very good chance
of surviving if it had made its software more portable. It's a good thing you
mention Franz, because they're almost as old as Symbolics, and still around,
largely due to the fact that their system is easily portable. They don't waste
immense amounts of engineering resources on supporting platforms, but dedicate
it to improving features and performance and more importantly responding to
customer needs.

"1) repeating the design of the past (with the same problems and maybe slight
improved. There has been lots of talk and little action. It's difficult
because we don't have the hackers with knowledge and time for that."

It's difficult to do because when you get down to it, Zetalisp sucked for
systems programming. Portable Standard Lisp, Interlisp, CMUCL/Allegro/Clozure
(anything with LAP and VOPs) provide a much better model for how to build a
Lisp runtime.

It's also difficult to do because people like to waste their time imagining
how awesome hardware type-checking on FPGAs would be and how the new Lisp OS
will have super-duper database-backed storage and network transparency and
etc. etc.

"2) doing a new design which gets rid of the limitations of past designs
(single user, single image, no security, ...). That's even more difficult."

The key is the namespace. Symbol identity needs to be preserved across
users/processes, symbol values don't. This is actually a much simpler problem
than you assume.

~~~
lispm
Zetalisp sucked for systems programming? That's a pretty strong word. Have you
ever used it at all?

I doubt that Allegro CL is easier to port than Open Genera.

------
cageface
There is no such thing as the "best" software environment, anymore than there
is the best car. It all depends on what you're trying to do and what your
priorities and constraints are.

~~~
regularfry
That's the dogma, but is there a reason for it to actually be true? All we
know for sure is that there isn't a clear winner _yet_.

~~~
gukjoon
Software environments, languages, and other such abstractions aren't there to
be perfect. They are there to help human beings manage absurd numbers of
machine instructions. Maybe when we fully understand the human mind, we'll
figure out the perfect way to map machine instructions to concepts that humans
can grok. Until then, we'll have to make somewhat subjective design decisions
in the face of uncertain constraints and priorities, which is certainly not
going to lead to a "best" solution.

------
rwmj
That page is giving a 403 error now. What am I missing?

