
Systems software research is still irrelevant - mbrubeck
https://trmm.net/Utah2014
======
bcantrill
Ugh, this old troll. Something that took me a while to figure out: when
someone says that "nothing interesting is happening any longer in
[discipline]", it is virtually always code for "I feel old." Yes, things
change and different periods of innovation are (of course) different, but it
is wrong to cast value judgement on one era over another. Speaking as an aging
technologist, I can say that it can be hard to resist this -- and in
particular, it's tempting to view one's late twenties and early thirties as a
Golden Age, and everything later as derivative and uninteresting. If it needs
to be said: for those who are making this claim, they are often projecting
their own lives onto their discipline, and may be confusing their own personal
Golden Age with a broader one that may or may not have existed.

My domain -- operating systems kernel development -- has been particularly
vulnerable to this (in part because it has become increasingly specialized)
and this is something that I have heard for my entire career. Of course, there
has been tons of innovation in the kernel over those two decades (and I've
been lucky enough to be close to a bunch of it) -- and no one now would
seriously go back to the systems from two decades ago. (If anyone disagrees,
they should kindly put their money where their mouth is and run only systems
software from 1994.)

So yes, systems software is still relevant -- and so are a lot of things that
might feel "done." And if you are an aging technologist like me (protected
class, baby!) and you feel tempted to tell these youngsters that there is
nothing new under the sun, please check yourself: yes, you should educate the
rising generation about the problems that have been solved -- but never go so
far as to say that there is no room left for new ideas. Aside from the fact
that it's a demoralizing thing to say to a younger technologist, history will
likely prove you wrong!

~~~
linksnapzz
Love the work you've done Bryan, but your response above does nothing to
engage the salient points made in the link && elsewhere. Yes, we have better
ways of getting performance metrics than truss, circa Solaris 2.5…and thank
you!

But that isn't what this is about.

~~~
bcantrill
Sorry, what salient points am I not engaging? The author of the linked piece
is decrying what they perceive as a lack of "OS research and innovation" \--
and I'm saying that OS innovation remains alive and well, if perhaps more
technical and less accessible than it once was. I can of course be more
concrete and rattle these things off -- but I feel safe in leaving software
systems advances since 2000 as an exercise to the reader...

~~~
linksnapzz
In the link, you can see a chart, showing the hardware capabilities of a high-
end workstations, sampled at roughly ten-year intervals. In twenty-five years,
hardware capability has changed dramatically, while the kind of software used
to get work done remains static (mostly).

Why is that? Would you make the chart differently?

The utah2K paper referred to in the link is admittedly (by the author) a
polemic, so readers must understand that the writer may be shading things in a
way more subjectively negative. Of course, that doesn't make it a trolling
attempt, unless you can think of a particular chip the author wanted to knock
off of a particular shoulder...

You acknowledge that OS innovation "may be more technical and less accessible
than it once was". Why is that? In what manner? I don't think that you and
Pike are using "OS research & innovation" to refer to all of the same things
at the same time.

Here (from utah2K):

 _" New operating systems today tend to be just ways of reimplementing Unix.
If they have a novel architecture -- and some do -- the first thing to build
is the Unix emulation layer.

How can operating systems research be relevant when the resulting operating
systems are all indistinguishable?

There was a claim in the late 1970s and early 1980s that Unix had killed
operating systems research because no one would try anything else. At the
time, I didn't believe it. Today, I grudgingly accept that the claim may be
true (Microsoft notwithstanding)."_

...I can't see why that isn't as true today as when it was written fourteen
years ago. You can say that our Unix systems have improved, but that's a
matter of degree, and not of type. We've also made great progress in breeding
lean pigs that grow to marketable size in record time...but no matter how fine
they are, they are still pigs, and aren't lobsters.

~~~
kev009
This is non-sequitur, mainstream hardware has only evolved as well so why
would there need to be a "forget everything you know" revolution? Conversely,
is hardware research dead?

The biggest change from 2000 is prevalence of NUMA, and this was even
available back then in SGI and Sequent/IBM systems. I don't understand why
systems software would have to substantially change until something like
memristors drastically change systems architecture.

For a while there has been less emphasis on individual systems and more effort
put into distributed systems.. first with Hadoop style clusters and we're now
breaking into Mesos style clusters that provide OS-like semantics and services
over a network of systems.

But there's a lot of good systems research going on today. Some of it is
additive, some subtractive, but it's exists and is important.

------
notacoward
A lot depends on how "systems software" is defined. Very few people have much
use for a whole new kernel. Even little pieces of new kernel _code_ are less
useful than before, as various functions (e.g. storage) can be handled just as
well in user space. So if "systems" == "kernel" then relevance really has been
declining. On the other hand, if one considers "system" to include user-space
storage and networking, distributed consensus and service discovery, package
and system management, JIT and GC in language runtimes, etc. etc. etc. then
we're actually in a bit of a renaissance.

The fact is that kernel/user hasn't been a very interesting boundary for a
long time. There are hundreds of systems-programming problems that can be
solved on either side of that boundary, except that if you try to solve them
in the kernel debugging will be harder and you'll get embroiled in the
constant turf wars that are the hallmark of a declining specialty. The idea
that only kernel hackers do systems programming needs to die.

~~~
jdimov
Does this mean we probably need a new, more minimalistic concept for a modern
kernel?

~~~
notacoward
We've been moving that way for quite some time, despite Linus's historic
dismissal of microkernel ideas. Granular capabilities, FUSE, namespaces,
cgroups, containers etc. all take us in that direction. Over the next several
years, I think even the kernel grognards will start to see that the "center of
gravity" for systems software has shifted to better languages and tools than
are available in their sandbox.

------
fasteo
Not sure to agree; there are quite a few interesting projects in both OS and
system programming language fields. To name a few:

OS

Mirage OS[1] and unikernels in general (ErlangOnXen, HalVM)

OSv [2]

Chromium OS [3]

Programming languages

Julia [4]

Rust [5]

D-Lang [6]

Nimrod [7]

[1] [http://www.openmirage.org/](http://www.openmirage.org/)

[2] [http://osv.io/](http://osv.io/)

[3] [http://www.chromium.org/chromium-os](http://www.chromium.org/chromium-os)

[4] [http://julialang.org/](http://julialang.org/)

[5] [http://www.rust-lang.org/](http://www.rust-lang.org/)

[6] [http://dlang.org/](http://dlang.org/)

[7] [http://nimrod-lang.org/](http://nimrod-lang.org/)

------
api
"There has been much talk about component architectures but only one true
success: Unix pipes. It should be possible to build interactive and
distributed applications from piece parts."

Java has many success stories in this area, starting with the amazing
reusability of POJOs (plain old Java objects). I can take Java code from 1996
and _almost always_ compile it and use it now without major refactoring and
without having to really look at the internals. I can do it with the API docs
only. Beyond POJOs there are many abstraction techniques like beans that have
been used to build huge software systems that have stood the test of time,
albeit with a bit of clunkiness. But Unix pipes are a bit clunky too.

I coded a lot with Java years ago. It's been out of vogue for quite some time
-- though it's still very popular just not trendy. Lately I've been reflecting
on some of the pain points of other languages and thinking "wow, Java really
did have a _lot_ of things right."

I've also been thinking that the JVM is _almost_ an OS, and that it sort of
wants to be. There were some efforts in that direction, but none really went
anywhere. It's not quite perfect, but I feel like it's the only thing we've
had in the past 30 years that looks like a step toward a post-C/Unix operating
system that could really work. I also feel like in the age of many-cores we
could do away with problems like GC pauses completely. I could forsee a future
many-core box where 2-4 (of 32-64) cores are just almost flat out dedicated to
GC, JITing, tracing, optimization, etc. When you've got that many cores you
could dedicate a few to the OS and not worry much about it. The benefits would
be things like forgetting about manual memory management, etc.

Also note that the JVM can run more than Java. Java would just be the "core
language" of this hypothetical OS, taking the place of C on Unix.

~~~
MrBuddyCasino
> I also feel like in the age of many-cores we could do away with problems
> like GC pauses completely.

Unfortunately its not a resources problem, or else there wouldn't be an issue
with GC pauses on fast hardware.

There is a solution for that though, and Cliff Click of Azul Systems is
willing to sell you the only pause-less JVM GC there is for the low low price
of 7500$ per server (see
[http://www.azulsystems.com/products/zing/whatisit](http://www.azulsystems.com/products/zing/whatisit)).

~~~
api
My guess is he's using lock-free data structures + a lot of cleverness.

~~~
hga
Somewhat more than that:
[http://www.azulsystems.com/sites/default/files/images/c4_pap...](http://www.azulsystems.com/sites/default/files/images/c4_paper_acm.pdf)

Including a read barrier variation and batch VM remapping operations (change a
bunch of large pages, only invalidating the TLB once at the end).

------
josephlord
Xen largely came from the Systems group at Cambridge I think. Seems fairly
relevant to the current state of computing to me. Not sure what is being
worked on now that might be relevant in a decade and a half from now [edit to
add: but there is probably something].

~~~
amirmc
> _Not sure what is being worked on now that might be relevant in a decade and
> a half from now._

Mirage OS is one of those things -
[http://openmirage.org](http://openmirage.org) \- though I expect (hope) it'll
take much less than decade to be relevant. Some of the same team that worked
on early Xen are working on Mirage.

~~~
pron
And there's OSv, [http://osv.io/](http://osv.io/), from the people behind KVM.

------
donavanm
Hrm. I think there are some decent xounter arguments. I will agree that pure
academic systems research is being superceded by industry development or
academic commercialization. As a trend I think commoditization, parallelism,
and management are where effort (and sucess) has gone, as opposed to novel
research. Make it cheaper, wider, and possible to manage.

Influential counterpoints: distributed systems
management/configuration/orchestration, distributed datastores (CAP, CALVIN,
Dynamo, etc), system languages (Go, Rust), distributed processing (tilera,
niagra, larrabe, gpus), network fabrics instead of trees, &stream processing
jnstead of batch (kinesis, millwheel).

~~~
Dewie
> , system languages (Go, Rust),

How is Go not another example of how "programming languages have stagnated"[0]
? You could argue that it is really practical and productive, but that is a
separate point to whether it or not it is "more of the same".

Or maybe it is more of the same, just in the vein of languages that most
people have forgotten about by now, so it is for all intents and purposes
_new_. Certainly native compilation and some control over memory layout (I
guess, with structs and such) and indirection have not been common features in
recent memory (that is; those two points _and_ automatic memory management,
_and_ overall simplicity compared to C/++).

[0] From the article:

> It included a chart lamenting the stagnation in [...] programming languages

------
Someone
Meanwhile, on the HN front page, we have:

[https://news.ycombinator.com/item?id=8651499](https://news.ycombinator.com/item?id=8651499):
_Kill init by touching a bunch of files_

[https://news.ycombinator.com/item?id=8650952](https://news.ycombinator.com/item?id=8650952):
_on Linux, 'less' can probably get you owned_

One can argue that the research needed to prevent those was done decades ago,
but this article doesn't make that argument.

We may have hit upon a 'good enough' solution that is so hard to displace that
it may never happen, but I think systems software research is very much
relevant. Someone will have to polish a more robust/secure OS (may already
exist in unpolished form, but possibly even must still be written) until it is
shiny enough to replace the C-based (or equivalent) mess we have now.

------
bjackman
I don't think I _entirely_ agree - as notacoward says I think we're
approaching a renaissance in certain areas.

\- Rust and Go are becoming extremely popular extremely quickly.

\- I think/hope we may soon see some sort of battle between OS container tech
(Docker/LXC/cgroups, FreeBSD Jails) vs unikernels (Mirage, OSv, Linux
unikernels) on paravirtualised platforms (Xen) as the full-weight-VM-running-
Debian model ages.

edit: Oooh look, an exciting relevant systems software paper!
[http://anil.recoil.org/papers/2013-asplos-
mirage.pdf](http://anil.recoil.org/papers/2013-asplos-mirage.pdf)

~~~
pron
> Rust and Go are becoming extremely popular extremely quickly.

Unfortunately, that's yet another SV bias. Rust has hardly any adoption, and
Go has moderate adoption and is growing nicely, but nothing close to
justifying "extremely".

~~~
erickt
To be fair, we in the rust community are telling people to wait for 1.0 unless
they are okay with constantly updating their code as we clean up the language
for our first release. It's fair though to speak about momentum. Comparing
Go's contributor list
([https://golang.org/CONTRIBUTORS](https://golang.org/CONTRIBUTORS)) with ours
([https://github.com/rust-
lang/rust/blob/master/AUTHORS.txt](https://github.com/rust-
lang/rust/blob/master/AUTHORS.txt)) has us with about 100 more than Go. I'm
not saying our community is better than Go's or anything, but I'm certainly
impressed a new language has been able to gain so much interest so quickly.
When I first started contributing a couple years ago, we had like 15 people in
our irc channel. Now we got over 500.

~~~
pron
Oh, I didn't mean to criticize either of those languages, just to point out
that SV developers are a tiny, tiny minority among the software community, and
their early adopter style is not shared by the industry at large. Switching
programming languages carries immense costs (probably in the billions of
dollars across the industry), and companies make it only when the benefits are
dramatic and incontrovertible. Most companies wouldn't even consider switching
a programming language more than once in a decade, and some would only
consider such a switch every 15 years or so.

------
codeulike
He seems to think progress would look like "a new wave of built-from-the-
ground-up operating systems sweeping away all the old ones every decade or
so". Say, if Plan 9 had become dominant in the 90s, would we by now be
replacing it with something else? What would everyone moving to new operating
systems every 10 years gain us?

There has been mind boggling progress since the 90s, just not in the direction
he expected.

------
justincormack
I think there is a revival of interest, as mass scale automation is changing
the way we use servers, and security issues are finally changing the
desktop/mobile. I found a lot of interest when I organized a conference [1]
which is tomorrow (videos will be available), which si encouraging.

[1] [https://operatingsystems.io/](https://operatingsystems.io/)

------
nhaehnle
The interesting thing about the table shown in the article is that even while
the author laments it, it actually illustrates a _good_ development: the table
has an "hourglass" structure.

There is a stable part (fundamental technology such as the kernel, TCP/IP
etc.) at the center which acts as an interface between the top part (hardware)
and the bottom part (languages).

All innovation has to be anchored to some core set of stable interfaces into
which the new innovations "plug in". The nature of OSs is such that they are
naturally part of these core sets. That may be frustrating for OS researchers,
but they'll have to learn to live with it.

------
mcguire
Overall, I agree with Pike in 2000 and this article today. Systems software
research has become stagnant and significantly derivative. The point isn't
that a new OS would come along and sweep everything away, the point is that
_nobody seems to be developing new OSs at all_. Sort-of.

In 2000 or so, systems research had turned almost entirely into server
performance measurement and tweaks. That trend seems to have changed, but
things like Xen and VMware are not really new. IBM invented virtualization
back in the middle ages; most of the more interesting things going on recently
have involved using the idea. Likewise, moving stuff from Plan 9, etc., into
Linux is cool (very, very cool), but it's not new, groundbreaking research.

That being said, there are a lot of nitpicks one can make with this article
and with Pike's original talk.

The processor speed comparison in the table is not terribly valid, in my
opinion. The more important number is front-side bus speed, which has not
changed anywhere near as much. (Front-side bus speed dominates most of
computing loads I see, and a great whopping chunk of the work in those areas
where it doesn't dominate, video games and data-heavy computing, is in
preventing the front-side bus from being the bottleneck.)

Pike was wrong about the language environment in 2000 and the table is wrong
today. Heck, he might have been wrong about 1990. Programming language
research in general and its specific applicability to systems (Hi, ATS!) has
hotted up a lot since the '80s and '90s.

Pike's comment, "There has been much talk about component architectures but
only one true success: Unix pipes. It should be possible to build interactive
and _distributed applications_ from piece parts," bothers me a lot. A great
deal of component research has been a dead-end in general (See that pile of
crap with CORBA and SOAP laying on top?) Further, I believe Pike _never really
understood distributed systems_. I have seen Pike complain that Nagel's
algorithm has no place in TCP/IP (although I haven't been able to find the
reference in years), but if you want a more recent example, see Go---everybody
likes CSP, right? Everyone except someone who has tried to implement a
synchronous rendezvous on a network.

Finally, Plan 9 didn't fail. But it also didn't have any chance to succeed, if
by success you mean taking over the world. It was a great research project,
but I'd believe making someone who didn't grow up at Bell Labs should be
against the Geneva Convention. (Anyone know when Plan 9 started supporting
color?) Plan 9 is my prima facie evidence that Not Invented Here syndrome was
invented at Bell Labs.

~~~
bcantrill
There's a lot that's wrong in here, but one thing that absolutely needs to be
called out:

 _...things like Xen and VMware are not really new. IBM invented
virtualization back in the middle ages_

Actually and in fact, what VMware did _was_ new -- to the point that prior to
this work, many reasonable people believed that x86 couldn't be virtualized
without unacceptable performance penalty. Please read Keith and Ole's
excellent ASPLOS paper[1] for an in-depth discussion of the novelty -- and the
citation for this technology when it won the ACM Software Systems Award[2]. If
you insist that VMware's binary translation work wasn't novel, please be
specific about how these peer-reviewed bodies got it wrong.

[1]
[https://www.vmware.com/pdf/asplos235_adams.pdf](https://www.vmware.com/pdf/asplos235_adams.pdf)

[2]
[http://awards.acm.org/award_winners/bugnion_3886645.cfm](http://awards.acm.org/award_winners/bugnion_3886645.cfm)

~~~
mcguire
The ACM citation indicates that virtualization was first explored in the
1960s; Adams and Agesen's paper cites a 1974 paper:

" _Popek and Goldberg 's 1974 paper [19] establishes three essential
characteristics for system software to be considered a VMM:[...] In 1974 a
particular VMM implementation style, trap-and-emulate, was so prevalent as to
be considered the only practical method for virtualization.... We review basic
obstacles to [trap-and-emulate] of the x86 architecture and explain how binary
translation (BT) overcomes the obstacles..._"

In other words, the contribution of that work was an implementation technique
for virtualization. In fact, that particular paper exists to compare binary
translation to trap-and-emulate hardware support that had recently been added
to the x86 architecture. Their conclusion is that sometimes one is better,
sometimes the other.

Look, I'm not saying binary translation wasn't new, or that it didn't change
the face of computing as she is known---which is what the award was given for:

" _VMware Workstation 1.0 spurred a shift to virtual-machine-based
architectures, allowing users to efficiently run multiple operating systems on
their desktops [, and allow] large-scale data-center operators [to] increase
the efficient and safe use of shared computational resources._ "

I'm not even arguing that it didn't open up cool new research areas; I'm going
to go play with unikernals as soon as I get a chance. (Even though _that_ is a
return to pre-timesharing systems.)

But we're still talking about using virtualization to run legacy operating
systems on modern hardware. Just like VM/370.

    
    
        init cms

