
What Is Systems Programming, Really? - wcrichton
http://willcrichton.net/notes/systems-programming/
======
antirez
My idea of system programming is that, other than the "near to the bare metal"
element, which may not be always true, has this quality of creating
infrastructures for other layers to use. A game 3D engine and a DNS server may
be both written in C++ and may use the same low level programming techniques
to achieve speed, but the fundamental difference is that one is just part of
an application program of some type, and the other a system that provides a
general service to other programs. So clearly it's technically possible to do
system programming in higher level languages, even if often this will require
anyway dealing with many system calls regardless of the language, both
explicitly or implicitly via abstractions.

~~~
zaarn
I think systems programming should be split in two, honestly.

You have the kernel-level systems programming where you need to be near the
bare metal, where you write kernels or program microcontrollers with less RAM
than a x86 CPU has L1 cache, where taking a microsecond longer can mean the
overall system crashing (or even costing a human life).

And you have system-level systems programming where you write services and
supportive infrastructure to enable user programs, where garbage collection
and non-realtime behaviour is acceptable, where you can skip a microsecond and
it'll be alright.

~~~
_dps
I think the two "poles" you've identified are good ones (~microseconds matter
and ~100ms matters) but there's a wide range in between. If you do anything
with video at modern resolutions, then you're in the single-millisecond-
matters regime (this includes games as mentioned upthread, and just about
anything in VR as well).

~~~
zaarn
Video streaming and gaming have some soft realtime guarantees, yeah, though
generally these aren't systems programming and application programming
instead.

The plumbing below, ie graphics rendering pipeline, tend to fall into the
category of kernel systems programming, controlling the GPU is pretty close to
doing what a kernel does IMO.

------
thegeomaster
Great article. This is an important distinction to make. I've noticed that the
more experienced I got, I felt less and less need to micro-optimize every
aspect of a program at the expense of code readability. That's where I think
the design and implementation of a system intersect; making it easy to
understand how it works and giving future maintainers the confidence to make
changes is best accomplished with clear and readable code with the right
abstractions (even if it hurts performance a little). I think that
documentation will always be second to that.

~~~
codr4
Depends on the kind of software, though. If you're writing an
interpreter/compiler, web server, database or operating system you don't have
much choice.

------
mrec
> _Andrei Alexandrescu (creator of D)_

Andrei didn't create D; he's been influential in it, especially w.r.t. D's
metaprogramming story, but the language was created and is primarily
maintained by Walter Bright.

~~~
bausshf
D1 was created by Walter Bright, but D2 was primarily created by both Walter
Bright and Andrei Alexandrescu.

~~~
mrec
Is D2 so different as to be considered a separate language? (I haven't really
followed it for years now.)

~~~
bausshf
Yes it's very different.

D2 introduced a lot of new keywords, new semantics, not two "standard"
libraries like with D1 where there were Tango. Tango is pretty much completely
obsolete in D2. Most D1 code might work to an extend with the D2 compiler, BUT
almost no D2 code will be able to run using a D1 compiler and it'll even be
difficult to translate it directly.

D2 is basically when Andrei joined the development of D back in 2007.

------
toolslive
> You should be able to forge a number into a pointer, since that’s how
> hardware works.

It's a C-centric view of the world and I wonder what old school FORTRAN77
people or lispers would think of this.

Anyway, I think the right to do this should be a privilege of the compiler. As
soon as you claim this right, you abandon all possible support she can give
you in battling all kinds of silly mistakes.

~~~
Athas
Pointers are not even necessary for writing operating systems or memory
allocators. The Oberon system writes those low-level components using
intrinsic peek/poke functions (like some ancient BASIC!) that are recognised
specially by the compiler and turned into direct memory modifications. This is
clearly just as unsafe as pointers (probably more so), but it means you don't
generally pollute the language to support some very specialised code. The rest
of Oberon is memory-safe and garbage-collected (with the garbage collector
also written in Oberon).

~~~
josteink
> The Oberon system writes those low-level components using intrinsic
> peek/poke functions (like some ancient BASIC!) that are recognised specially
> by the compiler and turned into direct memory modifications. This is clearly
> just as unsafe as pointers (probably more so), but it means you don't
> generally pollute the language to support some very specialised code.

How is "val = PEEK adr" and "POKE adr, val" any less polluting than allowing
"val = &adr" and "&adr = val" in select places?

~~~
pjmlp
It shouts to you, "Take care something dangerous going on".

Hieroglyphs are easy to pass by.

~~~
josteink
Lots of languages (like C#) allow pointers, but only inside blocks explicitly
declared to be unsafe.

    
    
        void DoSomeUnsafeStuffHere()
        {
            // regular code here. pointers verboten.
            
            unsafe
            {
                 // pointery stuff here
            }
        }
    

I think that is equally clear, if not even more. And again: I don't think this
is any more polluting than weird out of place PEEK/POKE statements.

~~~
pjmlp
That is true, my point was that for C and C++ that is not clear at all.

You cannot simply search for & and __* regarding point operations because
those are also valid numeric operators, thus with context dependent meaning.

Being able to search for PEEK/POKE, SYSTEM.PUT/SYSTEM.GET, or just unsafe
blocks makes all the difference tracking down unsafe code.

Also note that unsafe code blocks is older than C almost for 10 years, as
ESPOL/NEWP already had such feature in 1961.

~~~
Koshkin
> _You cannot simply search_

Which is one of the reasons why in the modern world (of fast CPUs and large
RAM) code should be edited using editors that "understand" the language's
syntax.

------
dleavitt
The author's point that functional programming principles are important to
systems programming and should be taught alongside it is well taken. I think
he confuses the issue somewhat with all the discussion of what is a "systems
programming language" as these days it's just used to refer to languages
substitutable for C.

------
jillesvangurp
System programming is definitely a bit overloaded these days though I believe
it is now widely understood to be languages like C/C++ that can be used for
OS, kernel, and embedded development.

I would agree though that the language ecosystem is shifting a lot in the last
few years. IMHO there are now a few languages that are becoming proper full
stack languages in the sense that they scale from embedded all the way to
browser development. Rust is a great example.

I'm really impressed with what the Rust community has done in just a few short
years. All my life, C/C++ was the only game in town for the stuff that is now
done with Rust with arguably the same or in some cases slightly better
performance.

Stuff like WASM allows the use of system programming languages in places where
they would not be used in the past. Using Rust transpiled to WASM makes it a
proper fullstack language. And people are using it for that. And people do
server development as well in it as well as OS development.

~~~
saagarjha
> IMHO there are now a few languages that are becoming proper full stack
> languages in the sense that they scale from embedded all the way to browser
> development.

I think you're being a bit optimistic here: traditionally, the only language
that has been able to do this is C++, and Rust is displacing it simply because
Rust really tries to target C++ developers and their pains. But other than
that I don't see much else development here. Swift has promise, but
unfortunately nobody's really writing much framework or foundational code with
it; at least not yet.

~~~
clouddrover
> _the only language that has been able to do this is C++_

I'd say Object Pascal is able do it too, though it's certainly true that
Pascal is not as widely used as C++. Free Pascal and Delphi are the two major
Pascal implementations today:

[https://www.freepascal.org/](https://www.freepascal.org/)

[https://www.embarcadero.com/products/delphi](https://www.embarcadero.com/products/delphi)

~~~
pjmlp
That Borland management board.... :(

------
n_t
Irrespective of language and framework, if the code needs to be aware of
underlying hardware then it is systems programming. ex. kernel code, user-
space drivers, databases, portions of cloud or server code which need to be
hardware aware, etc. What golang targets, I think, should be more of
middleware or service-level programming, whether its containers or servers.

~~~
steveklabnik
Alan Perlis has a thoughtful way of expressing this: “a programming language
is low level when its programs require attention to the irrelevant.”

At first I thought it was poking fun; now I see it as a comment about what is
relevant at a given level of abstraction.

~~~
parley
I like that line of thinking.

When discussing the suitability of different programming languages I always
point to the problem at hand and identifying the abstraction level the problem
is at. What your problem requires you to care about and pay attention to gives
this.

Ideally, solving your problem would be a one-liner in an already existing DSL
created for your specific problem (domain). In reality, this rarely happens
and you must have your pick beteeen everything from niche DSL:s to assembly,
but the key really is identifying that problem abstraction level.

I work in a mainly C/JS shop whereas I privately prefer Rust/ReasonML. Rust
would be a great fit at work, but Go would also work nicely for tons of things
we do. Alas, the inertia of organizations.

~~~
ModernMech
> Ideally, solving your problem would be a one-liner in an already existing
> DSL created for your specific problem

I like thinking about the practice of programming as creating a DSL in a
language to solve your specific problem.

------
jkingsbery
I'm curious where Erlang fits into this story. It was built to be a "systems"
language but algorithms are tersely implemented (I forget exactly, but I
remember reading about a study that an Erlang program is much shorter than the
comparable C program), and has gradual typing of a sort. It also is garbage
collected (I don't think you could "write your own memory allocator in it", as
Andrei Alexandrescu described).

------
RickJWagner
Nice article.

My view of Systems Programming will forever be colored by my start with IBM
mainframes. The Systems Programmer maintained all the utilities that the
mainframe provided, and also the frameworks that supported online programming.

I suppose today's equivalent is a combination of a sysadmin and an
applications developer, providing the applications are things like container
components, build pipelines, etc.

~~~
vram11
They call them devops these days.

------
burfog
A systems programming language can run on bare hardware by itself, or nearly
so. It is acceptable to require a very small amount of assembly code, for
example to implement something like memcpy or bcopy, or to provide atomic
operations.

A language is disqualified if it requires an OS or if it requires code written
in a different non-assembly language. Cheating, by adding that as a huge
(impractical) amount of assembly, also disqualifies a language.

Funny story about gcc: The compiler demands a memcpy, which it sometimes uses
for struct assignment. If you implement memcpy in the normal way, gcc will
helpfully replace your implementation with a call to memcpy! Despite this, C
is still a systems programming language.

~~~
tralarpa
> It is acceptable to require a very small amount of assembly code, for
> example to implement something like memcpy or bcopy, or to provide atomic
> operations.

There is some stuff missing in your list: preparing the stack pointer, address
layout, etc. You also need tools to produce text and data sections that can be
loaded at a specific address. Even if C is a low level language, there is
still quite some stuff between it and the "bare hardware". In that sense, the
only language that gives you enough control to produce the binary exactly in
the form needed by the hardware (without relying on external tools or
libraries) is assembly.

~~~
saagarjha
> You also need tools to produce text and data sections that can be loaded at
> a specific address.

Though, of course, most compilers provide extensions that make this task
generally possible within C (using the loose definition of "I don't need any
separate files or inline assembly, just attributes and flags).

~~~
raggi
So we're getting closer to it. The linker inputs and scripts are the only
"real" systems programming language.

------
MichaelMoser123
I don't think a single definition makes any sense now: decades ago you may
have had a definition of a 'computer system' that describes every big program.
Nowadays there are many sets of requirements depending on context, each set of
requirements implies its own constraints on memory management and scheduling
so that there is no one description of how a complex program would look like.

------
boulos
You’re HN famous, Will!

But I disagree with this:

> Companies like Dropbox were able to build surprisingly large and scalable
> systems on just Python.

Many of Dropbox’s _services_ are written in Go, and Magic Pocket is written
(partly, at least) in Rust. Earlier in their development, Dropbox relied on
S3, which is obviously not in Python.

Fundamentally, I think the important part of the “production” aspect of
“systems programming” is that it’ll be used a lot. That’s what drives the
requirement for efficiency: if you’re 10-100x less efficient, someone else
will be more cost effective / solve bigger problems.

As an additional example, GitHub was written in Ruby, but that’s fine because
the underlying Git and filesystem manipulation is all in C. The same thing is
sort of true of Facebook’s world of PHP: PHP is mostly a wrapper around C
libraries. Until it got complex enough that they rewrote the language.

tl;dr: I don’t think systems programming needs to be “low level”, but
production systems do need to be efficient. More powerful computing just moves
that further to the right!

------
sytelus
In old days, computers that came from manufacturers like IBM, DCE was refrred
to as system. The operating software for the system was referred to as “system
software”. People who created that software were called “systems programmer”.
Much of it invariably involved writing direct to metal/assembly code which is
why systems programming has been associated with low level programming. People
using this term to refer to task of doing some complex application
architecture are doing it wrong.

------
dfox
I think that it is not useful to build on Ousterhout's definition as his
definition of systems programming language is mostly meant as contrast to Tcl
with it's everything is a string memory model (in that view one could very
well conclude that Python is a systems programming language).

------
theoldgit
when I was programming in the 70s and 80s (in the UK), the term systems
programmer applied to those people involved in the configuration, tuning,
management of software such as CICS, MVS, and their equivalents on other
hardware bases. I worked at Sperry Univac for several years and the "systems
programmers" were the engineers who tweaked the OS, the TP schedulers, built
the product installation scripts, designed the configs that controlled the
software. It was a specific tradecraft suited to those who were more
technically focused, remember that COBOL was the prevalent language for nearly
all programming roles in the 70s and that there was no "web" per se. The term
started to fade away roughly when PCs started entering into mainstream
computing

------
kmooney
If you are making system calls–without too much abstraction–you are doing
systems programming.

------
tempodox
I think the author makes fair points. My conclusion: Do the bit twiddling and
strongly constrained parts in a low level language and the rest in a language
that supports strong static typing and good interfaces.

------
gjvc
... this deserves a supplementary question: what will get us beyond UNIX to
the next platform, not just the next paystub?

~~~
pjmlp
Maybe something like Android, OS X/iOS, ChromeOS, UWP, Fuchsia, Unikernels,
Language runtimes on Hypervisors.

And since I sense the question coming, Android, OS X/iOS, ChromeOS might use
an UNIXy kernel, but its exposure surface to applications is so small, that it
can be easily exchange for something else, it is just a matter of cost.

Which brings us to the next point, given the commodity of free UNIX-like OSes,
maybe only an hardware reboot like Quantum computers will actually do it.

~~~
saagarjha
> And since I sense the question coming, Android, OS X/iOS, ChromeOS might use
> an UNIXy kernel, but its exposure surface to applications is so small, that
> it can be easily exchange for something else, it is just a matter of cost.

Android, macOS, and iOS expose almost the full set of POSIX to the programmer.
Now, you may argue that it goes mostly unused, but it _is_ there, and there
are people using it.

~~~
pjmlp
Android does not, use at your own peril as it is not part of the official APIs
and will get your app terminated if you use unauthorized APIs.

Using libc is not the same as POSIX.

[https://developer.android.com/ndk/guides/stable_apis](https://developer.android.com/ndk/guides/stable_apis)

[https://developer.android.com/about/versions/nougat/android-...](https://developer.android.com/about/versions/nougat/android-7.0-changes#ndk)

[https://android-developers.googleblog.com/2016/06/android-
ch...](https://android-developers.googleblog.com/2016/06/android-changes-for-
ndk-developers.html)

As for macOS, and iOS, if Apple removed POSIX how many apps written in
Objective-C and Swift (not UNIX cli tools ported from Linux) would actually be
affected?

Very few, because those written in C and C++ ported from other platforms are
most likely making use of ANSI C and C++ standards.

------
tanilama
Nowadays it broadly means using C++ programming, involving dealing with
different levels of system calls.

------
matachuan
IDK about systems programming, but one thing I do know: Systems researchers =
^[AI researchers]

------
danmg
640x480, 16 colours, Ring 0.

------
chubot
Hm I really like the observation that "systems programming" is overloaded.

I gave my take here:

[https://lobste.rs/s/jrzwgy/what_is_systems_programming_reall...](https://lobste.rs/s/jrzwgy/what_is_systems_programming_really#c_ubvdwf)

In short, I think _shell_ is actually a more appropriate language for
describing systems, rather than OCaml or Haskell! If that sounds weird, then
I'll point you to the evidence of shell already being used for this purpose
(e.g. the 40K lines of shell in the Kubernetes repo I mentioned).

Although, on reading the blog post a little more closely, we may not be
agreeing precisely on the problem.

I should probably write my own blog post about this ... there are a couple I
linked that give something a flavor.

~~~
v_lisivka
It's simple:

Bugs in application code causes errors in that application.

Bugs in system code causes errors in an user application.

If bugs in your code causes problems only to your code - it's single
application. If bugs in your code causes problems in other programs - it's
system of programs.

------
qubax
In one word - syscalls. What do I win?

