
Systems Programming in C# by Joe Duffy - pjmlp
https://www.infoq.com/news/2016/06/systems-programming-qcon
======
twotwotwo
It always seems like a loss to me Joe's blogs read he's still selling the
Midori C# OS to some extent, and either sees no real downsides to it or
doesn't want to talk about them. Like, on the page introducing the series, he
says his biggest regret is it wasn't out there so "the meritocracy of the
Internet could judge its pieces appropriately," as if perhaps the big problem
was just misjudgment in a non-meritocracy. (And, hey, the Internet's judgment
sure has its warts too.)

It's fine to try to salvage good ideas from a project that failed at its
initial goals (and MS seems to have!), or even to hold on to thinking the
basic ideas were good even though the implementation failed. But if he could
just be candid about what _wasn 't_ right, he could spread those expensive
insights you get by actually building a thing, and show a real ability to
learn from failures. Who wants a postmortem that doesn't discuss what went
wrong? How useful is it to talk architecture if you aren't clear about
downsides and tradeoffs?

Concretely: What did the decision-makers who finally canned Midori dislike?
What were the performance bottlenecks (and numbers) after all their tuning?
Had they given up too much safety by the time they had it performing well? Was
it a compatibility thing, and if so what are his thoughts on an approach to
that? Was it just going to take too much investment to finish? I think a
little candor about the limitations, downsides, and hard-to-swallow parts of
Midori could advance the thinking about its basic ideas much more than a lot
of posts about implementation tricks.

~~~
pjmlp
Looking at the history of similar fate in other safe stacks, the political
reasons or desire to cut research money are higher on the list than technical
issues.

Everyone that was able to use Oberon or the Xerox stacks knows that it is
possible to have such systems and do productive work on them.

------
youdontknowtho
The first couple of comments are missing the point about this. Joe actually
built a real operating system with a variation of C#. Some of the lessons they
learned will be informing the next version of C#.

Mads has said that the changes that are coming are for apps like games that
need more low level features.

No one is saying that a previous version, the current version, or even the
next version of C# will be used to write an OS.

That being said, I really wish they had open sourced midori so that work was
available to build on. I know that a lot of you think that only C should be
used for OS development. (With the exception of a few of you that think Rust
is magic.) They actually built a real operating system with managed code. It's
basis in singularity included really novel ideas about where process
boundaries should be drawn and how an OS should be composed. It's a shame that
it isn't available. Not everyone wants to use or work on a UNIX clone.

~~~
pjmlp
Those ideas weren't novel in Singularity, just C's revisionism makes sure not
everyone delves into the history of system programming languages.

Check Burroughs developed in 1961.

[http://www.smecc.org/The%20Architecture%20%20of%20the%20Burr...](http://www.smecc.org/The%20Architecture%20%20of%20the%20Burroughs%20B-5000.htm)

Or read about the Mesa/Cedar Workstation at Xerox PARC:

[https://archive.org/details/bitsavers_xerox](https://archive.org/details/bitsavers_xerox)

If you want to read about a OS written in a fully memory safe language,
including the source code, check Niklaus Wirth books:

[http://www.ethoberon.ethz.ch/books.html](http://www.ethoberon.ethz.ch/books.html)

Specially Project Oberon. The 2013 re-edition has an updated hardware design
for an FPGA.

[http://people.inf.ethz.ch/wirth/ProjectOberon/index.html](http://people.inf.ethz.ch/wirth/ProjectOberon/index.html)

This is how it used to look like in its latest incarnations as BlueBottle
written in Active Oberon:

[http://www.progtools.org/article.php?name=oberon&section=com...](http://www.progtools.org/article.php?name=oberon&section=compilers&type=tutorial)

------
somethingsimple
I work on a project that does a lot of systems programming in C#. It's awful.
Myself and several coworkers every now and then ask, "why wasn't this done in
C or C++ in the first place?" So much jumping through hoops avoiding the GC
kicking in, or the GC moving memory around, doing P/Invokes, translating
things between managed and native, and so on... It's not fun at all.

~~~
curiousDog
Yup, same experience for us. We went back and forth between workstation and
server mode gc. If you have a process that maintains things like
leases/heartbeats in a low latency setting, C# doesn't seem like a good idea.
Wonder how well Go works in this scenario considering it was purpose built for
this.

~~~
nine_k
Why Go, it has GC, too. I'd look at Rust.

~~~
pjmlp
Because it is the spiritual successor of Oberon.

[https://en.wikipedia.org/wiki/Oberon_(operating_system)](https://en.wikipedia.org/wiki/Oberon_\(operating_system\))

[http://www.ocp.inf.ethz.ch/wiki/Documentation/Language](http://www.ocp.inf.ethz.ch/wiki/Documentation/Language)

~~~
nine_k
Oberon the OS was cool, but Oberon the language is sort of too 1990s. (But I'd
definitely take Modula-2 for low-level stuff instead of C any day, as I did in
early 1990s.)

~~~
pjmlp
I agree, hence why after my initial interest in Go I eventually switched focus
to other languages.

However, it doesn't change the fact that it allows for lots of low level stuff
in the similar vein as Oberon, which is why I happen to take Go's side, even
if I rather use other more expressive programming languages.

And to be faire, Niklaus Wirth latest language changes (Oberon-07) are even
more minimalist than Go's.

------
vmarsy
Very interesting! No matter what the original intent was with Midory, writing
an OS in C# and gain insights into what is wrong in C# Performance-wise is
great. All of these findings can have impact on every C# program out there: a
web server isn't considered "Systems programming", but it's not going to say
NO to performance improvements

In the slides posted by pjmlp[0], I found one slide particularly interesting:
Slide 38 about Contracts:

    
    
       Contract.Requires(buffer != null);
       Contract.Requires( 
         Range.IsValid(index, count, buffer.Length)
       );
    

or the Debug variant of it :

    
    
      Contract.Debug.Requires/Assert/Fail
    

It reminds me of the Dafny programming language[1], but here this seems to be
used for performance. The future C# AoT compiler could be validating those
Contracts, and from these Contracts enables more aggressive optimizations

The slide about PackN (and future "Safe" stackallock) is also great, it seems
like the easiest optimization someone can apply to its current code.:

    
    
      int[] array = new int[8] { 0, ..., 7 }; 
      // Heap allocation!  For short-lived arrays, this is bad!

Versus the proposed "Safe":

    
    
      Span<int> span = stackalloc int[8] { 0, ..., 7 };
    

[0] [https://qconnewyork.com/system/files/presentation-
slides/csy...](https://qconnewyork.com/system/files/presentation-
slides/csysprog-qcon-june2016.pdf)

[1] [http://research.microsoft.com/en-
us/projects/dafny/](http://research.microsoft.com/en-us/projects/dafny/)

~~~
EpicEng
>writing an OS in C# and gain insights into what is wrong in C# Performance-
wise is great

In some ways, yes, but was C# ever intended to be a systems language? No, and
that's obvious from its design. So, what is this _really_ telling us? That the
language has issues when used in a way it was never intended to be used?

~~~
pjmlp
In the 90's C programmers used to state that C++ wasn't a systems programming
language, now their main FOSS compilers are written in C++.

~~~
nickpsecurity
On top of that, even some L4 microkernels are written in C++. So, efficient
that they fit into a L1 cache as they run ulta-fast. I keep trying to get
people to _not_ use C++ for these but funny it's almost a trend now.

~~~
naasking
> I keep trying to get people to not use C++ for these but funny it's almost a
> trend now.

I can understand why L4 choose C++ at the time, as it had stricter checking
than C, but OS kernels have very little internal code reuse that necessitates
inheritance or templates. This is doubly true of microkernels. There is
literally no reason to use C++ in this domain. Ada or C, and soon Rust should
be the only considerations IMO.

~~~
nickpsecurity
Total agreement. :)

------
pjmlp
Slides are available at [https://qconnewyork.com/system/files/presentation-
slides/csy...](https://qconnewyork.com/system/files/presentation-
slides/csysprog-qcon-june2016.pdf).

Usually the videos take some time to appear at InfoQ.

~~~
atesti
The slides link to
[https://github.com/joeduffy/csysprog](https://github.com/joeduffy/csysprog),
but this has been deleted. Does anyone have a copy or fork? Thanks!

~~~
pjmlp
Just tried it now and the link is working.

~~~
atesti
Are you somehow logged in? I only see the github 404 for that link!

~~~
pjmlp
Sorry I misunderstood you, I was thinking about the PDF link.

You are right about the github one.

------
amaks
It's frustrating to see that people advocate to use programming languages in
the areas where they don't belong just because those languages are convenient,
have nice syntax, popular, etc. Go for example started that way but ultimately
went to the niche of writing networking services where it shines. It's weird
that languages with unpredictable runtime characteristics such as Java and C#
are being advertised as systems programming languages.

~~~
xoluxo123
You realize that Go also is GCed right?

~~~
amaks
Yes, and I never said that Go is a systems language. I said Go has found a
niche of networking services. This doesn't make it a systems language.

~~~
pjmlp
It makes it, because it has all the same language features as Oberon has.

[https://en.wikipedia.org/wiki/Oberon_(operating_system)](https://en.wikipedia.org/wiki/Oberon_\(operating_system\))

[http://wiki.osdev.org/Go_Bare_Bones](http://wiki.osdev.org/Go_Bare_Bones)

If I am able to write an OS using just the language, with the help of some
Assembly, or bootstrap the language and runtime, it is a systems language.

Many of the criteria people use to judge systems languages like inline
assembly, would disqualify C when applied to a pure ANSI C compliant compiler
without language extensions.

~~~
amaks
The primary criteria where a system language should be judged is the control
over the underlying machine code execution. With GC languages you don't have
any - the GC will kick in unpredictably. It may not be an issue for UI
(although everyone hates when UI stumbles), but for system code like OS, DBMS,
etc. it's simply not acceptable. Languages like C and Rust offer this level of
control - you simply know exactly what your code is doing at any given time.
With JVM, .NET, Go - you don't.

~~~
pjmlp
And yet Xerox PARC, UK Royal Navy, DEC all managed to write operating systems
in GC enabled systems programming languages, some of them quite productive
Workstation OSes.

You don't want the GC to mess with your lovely data structure?

Allocate on the stack, as a global static or let the GC know not to touch it.

Check Algol-68RS, Mesa/Cedar, Modula-2+, Modula-3, Oberon, Oberon-2, Active
Oberon, Oberon-07.

