
The state of low-level system programming - bluehex
http://erik-poupaert.blogspot.com/2012/12/the-state-of-low-level-system.html
======
pcwalton
Freestanding, ABI-level interoperability with C is planned for Rust, but is
not yet implemented. There are two main reasons why being able to be called
with dlsym() was less important for us than the current setup:

1\. M:N scheduling. We wanted high scalability with hundreds of thousands or
millions of tasks, which means we need the capability to have our own
scheduler. OS threads aren't lightweight enough for this, especially on
Windows, where over 80% of our Firefox users are. Having our own scheduler
means we need our own ABI. We would like to make the scheduler optional, but
getting it working was a higher priority than making it optional.

2\. Safety. On many platforms (basically, anything but MSVC on Windows), the C
stack is not bounds-checked; either there is no guard page or the guard page
is unreliable because you can overshoot it. This is not acceptable for our
Rust code, since stack overflows are a frequent source of exploitable security
vulnerabilities for us. So we do our own stack management with a custom ABI
that performs stack checks at every function entry, freeing us from this
source of pain on all operating systems. Again, this is planned to be
optional, but it was more important at this early stage for us to demonstrate
advantages over C than for us to duplicate its ABI exactly.

Many libraries, however, would prefer direct interoperability with C to M:N
scheduling or stack overflow safety. So we plan to provide a mechanism for
Rust code to be called from C without having to set up a scheduler or perform
stack checks. It's not implemented yet, but it's very much part of the current
design, and I would be thrilled to see it happen.

------
tehwalrus
I asked a very relevant question on stackoverflow about this recently:
"Writing a Python extension in Go"
[http://stackoverflow.com/questions/12443203/writing-a-
python...](http://stackoverflow.com/questions/12443203/writing-a-python-
extension-in-go-golang)

Now technically you can compile a _special_ python which can handle Go
extensions, which is the wrapper mentioned in the second answer, but you can't
distribute python modules that ask people to install "this special python" to
run (which will reliably work with all their other installed code..)

I would like to see a "go init" function in a C API, that would let you
interop the two whichever had the `main()`, since that would let you link with
all the other c-implemented scripting languages (python, ruby, etc) which
would mean that Go extension support could be added to the main cpython trunk
as a compile-time option.

------
bluehex
I really hope to see Go and Rust move in the direction of pluggable runtimes
that can be embedded into a C host program. Currently, both runtimes require
being seated as the main entry-point of the program to function. Both projects
also have issues in their bug trackers to rectify this:

Rust issue: <https://github.com/mozilla/rust/issues/3608>

Go issue: <http://code.google.com/p/go/issues/detail?id=2790>

I wonder who will get there first.

~~~
PuerkitoBio
Also, a way to compile without kernel (system) calls, so that it can be use
for kernel development. It used to be possible with go, but no more with go1.
Then these languages would truly be able to replace C.

------
groovy2shoes
Clay[1] doesn't get as much attention as it deserves. The authors seem well-
versed in programming language theory, but they also understand the practical
needs of a systems language. What you end up with is a language that's
sufficiently low-level to be a C replacement, but it offers a plethora of
mechanisms for abstraction.

Of all the new languages billed as "systems languages", Clay is the only one
that excites me.

It can also "quack like a duck" (to put it in TFA terms).

[1] <http://claylabs.com/clay/>

~~~
MatthewPhillips
I can't find any kind of getting started guide, and most of the libs have no
documentation at all.

~~~
qznc
I found a "Clay for C++ programmers" tutorial

[https://github.com/jckarter/clay/wiki/Clay-for-C---
programme...](https://github.com/jckarter/clay/wiki/Clay-for-C---programmers)

------
cturner
I don't think this stands up: "It must be possible for such contender to be
called from C code, because that is what it takes to be called from a modern
scripting engine too."

Later the author says, "If programmers wanted a statically-typed, compiled
system language for their scripting efforts, they would be using the existing
system languages for that already."

I think the reason that scripting languages exist is because of limitations in
the established languages. What we think of as "a scripting language" is
defined by the gap. If someone else could be a system programming language
without the gap - maybe it doesn't need to be accessed.

The traditional unix platform is (1) C and a compiler for it, with access to
the unix standard library and (2) some sh or csh derivative, and access to the
likes of sed and awk.

Shell gave people a glimpse of what was possible. But shell has poor access to
the unix API, and hits the wall quickly once you do anything other than very
basic tool development.

So these things called scripting languages started to pop up. People wanted
easy access to the unix API like C. They want easy string manipulation like
shell. And they want access to functions, maps and lists without having to
think about memory allocation.

If you start from the shell and awk and work into that gap you get perl - the
canonical scripting languge.

TCL inhabits a similar space to perl. It's also "a scripting language". But it
started at the opposite end. TCL starts as an extension language for C.

So I can understand the author's emphsais on close interaction with C. But I
don't think they're important if you can take away the shortfalls they're
compensating for.

I hope Go will mean I don't have to think about these things. That I'll be
able to get a standard build and a go compiler/runtime and have immediate
access to lots of power. No more headaches from C build tools or reaching
around in the murky water underneath my scripting language.

~~~
jedbrown
You are leaving out the importance of libraries and coupling of systems that
were initially written independently. This is the same mistake that many Lisps
have made, assuming that just because the technology is "superior", people
will flock to it and eagerly rewrite all their code (starting with their
dependencies).

------
justincormack
The C ABI is not great for interacting with scripting languages though, even
if it is the best we have right now. For a start there is no type information
obtained with dlsym and you basically have to parse headers the interpretation
of which is not clearly defined as the C language really defines an API and
the ABI is defined by the compiler. Many libraries for this reason do not
define stable ABIs and are a pain to use from a scripting language. Hence
Python has two ffi modes an ABI interface and a compiler mode.

Now if someone produced a better solution to that it would be useful...

~~~
angersock
You know, it's strange, because the Lua folks seem to be getting along just
fine.

~~~
justincormack
Yeah but us Lua folks are picky about the sort of C we like to interface to.

------
zem
there's also ats [<http://www.ats-lang.org/>]. somewhere on my long-term todo
list is to write a library of efficient data strucures in ats, to take
advantage of some of its safety guarantees.

~~~
doublec
It's a pity ATS doesn't get more attention in the "Systems programming
language" field. It can call into C, be called from C, provides a powerful
type system, and can be used to write very low level code. I've written a few
posts about how ATS can interact with C on my weblog here:
<http://www.bluishcoder.co.nz/tags/ats/>

~~~
erichocean
I like reading your blog posts on ATS, but I don't think I could use it
effectively.

I've had much better luck with exhaustive testing with Klee and related
libraries (e.g. for testing multi-threaded programs).

For whatever reason, they map better to how I think about problems, and I've
found it easier to develop new code that is testable within those constraints.

~~~
doublec
If you start off treating ATS like an ML variant interfacing with C without
using any of the type annotations for safety you can go far. Then you can
tighten the types up for safety as you get more experienced or find you need
it.

You could write tests as needed and when a test fails write types that make
that failure as close to compile time as possible.

Another approach is to just write C code, either embedded in ATS files or in C
files that are linked into ATS. Then slowly convert C to ATS as you feel
comfortable.

------
gmt2027
Vala does not seem to get much attention. It borrows C# syntax (and maybe
Java), compiles down to C and offers nearly symmetrical interoperability as a
result. I believe Ubuntu's Unity is written in Vala. Knowing Java, it was easy
to acquire the syntax from the tutorial
[<https://live.gnome.org/Vala/Tutorial>] in very little time.

For transparent access to compiled C libraries, external C functions need to
be declared in a 'vapi' file and although these were already provided for a
large part of the GNOME ecosystem, documentation on vapi syntax was nearly
non-existent. Object-orientation and standard libraries are integrated with
Gnome's GObject system and Glib libraries which I found unfamiliar and usually
better documented in C at the time.

~~~
qxcv
If it has GObject as a dependency, then it's not designed for "low-level
systems programming". Hell, just having a GC more-or-less discounts it due to
the associated problems with runtime size and performance[0]. As a general
rule of thumb: if you can't write a kernel in it, it's not a systems language.

[0] It _is_ possible to write a systems language with these features, but low-
level systems stuff is not what Vala was designed for.

~~~
gmt2027
GObject itself is not a dependency, Inheriting from GLib.GObject provides
support for features like introspection and better integration with the
collections framework and GLib libraries. However, lightweight 'non-object
classes' are supported but treated differently by the compiler. There is a
slight variation in constructor syntax between plain objects and GObjects.

As far as I know there is no Vala runtime or garbage collector. Memory
management is by automatic reference counting but there is support for manual
C++-style pointer syntax with new and delete operators.

I agree that Vala was not specifically designed for 'low-level systems stuff'
but I see no reason why this is not technically possible. The Vala compiler is
a source-to-source translator to C and in my previous experience worked fairly
well interfacing with fairly low-level C libraries like MPI and OpenCL. I
assume that a kernel writer does not require much more than a limited subset
of the language that supports C pointers, structs and arrays.

------
dschiptsov
This is called FFI.)

And yes, a language with which you cannot dload and call any shared library
you wish is, how to say, handicapped language.

NodeJS, Java to name a few. Look what they do due to inability to call, say,
libpq.so.)

But of course, re-monkey-patching everything from scratch by amateurs coders
is much better idea. And, you know, you could create an entire "standard" with
hundreds of pages of specification and then push your bloatware - JDBC.

~~~
pjscott
I believe his thesis was actually that most languages are handicapped because
_they_ can not easily be called via the usual "load a .so file and call a
function" ABI. Java, for instance, would be a more useful language to me if I
could easily create Python bindings for useful Java libraries like Lucene,
without needing to use something like Jython that runs on the JVM.

~~~
dschiptsov
The confusion arises when some language developers decide to ignore or break
existing ABI for a particular platform.

There are general ways to call procedures - there machine instructions for
this. ABI defines how exactly parameters and return values should be passed -
using, registers or stack or both.

As long as you follow the rules there is no difficulty in calling everything
you wish.

The whole idea of JVM as something disconnected from reality^W OS and hardware
is simply wrong, and all the confusion is the consequence of that premature
decision.

Of course, slogan "you don't have to know" always wins.

------
rpearl
C code _can_ call into rust: <http://static.rust-
lang.org/doc/rust.html#extern-functions>

~~~
bluehex
True, and so can C call back into go code with cgo. The problem is that it
still requires that the main function be in a Rust/Go binary. A program
written in C can't currently call into a library written in one of these two
languages since the runtimes don't work unless they control the main entry
point.

From the Rust doc linked: "The primary motivation of extern functions is to
create callbacks for foreign functions that expect to receive function
pointers."

~~~
jzwinck
Then is it even possible to make a program that can use libraries written in
Rust and Go and C++? If everyone wants to control main(), is there an escape
hatch provided by any of these languages, like a C function that can be
explicitly invoked near program start?

~~~
smosher
You can do this with C++ already. Rust's plan seems to be to eliminate or
reduce its runtime and give up the entry point. Go doesn't appear to have a
plan.

------
MatthewPhillips
So what's the catch when it comes to D? If I can use all C libs easily, and
get some more palpable syntax, what am I missing exactly and why haven't
people transitioned to it?

~~~
pretoriusB
The article is not about using C libs, is about C programs using you (your D
code, that is).

That said, the problem with D: not enough marketing plus the two compilers / 2
libraries issue.

And it's close to C++ complexity enough to not let the scripting guys feel
welcome (like they do in Go).

------
zanny
Compatability with the C ABI I feel isn't a necesssity. On platforms built on
C it is, but if you wanted to invent a new machine running a new system
language with the ABI messes of C exonerated, you could. You would need to
devise a translation layer in the OS executor to be able to interpret classic
C ABI .so or .dll files for legacy applications and so you could compile C on
these platforms, but that isn't a prohibitively hard challenge.

~~~
sparkie
The need for C isn't so much restricted by the operating system, but by the
hardware manufacturers. The simple fact is that C is the defacto choice of
language for hardware vendors to ship a compiler for, and that doesn't look to
change any time. It's a standard that can't be avoided, unless we can work
cooperatively with enough major vendors to change this.

~~~
TheCondor
There is C specific hardware?!?

This is a myth and part of the vicious circle here. I'm not a C hater by any
means but the reason hardware vendors have historically included it is because
there are free compilers that are nearly C enough that they can include and
they have to include something. C is just a portable assembly, you can almost
write a simplistic compiler in pure yacc and more than a couple of these C
compilers hardware vendors produced weren't much more than that. Software guys
then wrote in C and the circle continues, intel has a great C compiler now and
in fact all the best optimizers are from hardware vendors but that wasn't
always the case. There was also a time (quite possibly it's still that time)
when you'd buy some customed up MIPS or PowerPC chip and get an ancient
version of gcc that they'd hacked to support their shit and forward porting to
a modern GCC wasn't always a straight forward thing to do. They have to
provide something and tools usually isn't their business.

Now so much is built on the C non-runtime that Go, Ada, Rust, anything with
any runtime looks completely backwards to a lot of system folk. What's funny,
you look at Linux and there is a fairly small hardware dependent directory of
code where the register poking happens, it's really not that much code but its
called from all the drivers and such, and that's the holdup. It's not like
that stuff makes up 50% of the code or anything.

~~~
Yttrill
Of course there is C specific hardware!

X86 provided efficient call/return protocol instructions including display
management, specifically targeting Pascal/Modula style languages. Microsoft
Windows 3.1 even mandated this calling protocol. However old style C didn't
use this API because it doesn't work with varargs and C doesn't have nested
functions.

Instead Intel observed real code and optimised their instruction set to speed
it up, locking everyone into lame languages like C. Attempts by Intel to break
from this, for example with x64, have failed (x64 has two stacks).

Similarly segmented architectures, which implement extension of the Harvard
machine model, have been largely unsupported in favour of linear addressing
von Neumann machines, primarily because that's what C and Unix use.
Interesting to see Microsoft adapt their OS designs based on this trend.

To support more advanced features in a language often requires only a few very
low level primitives. This is why Linux kernel doesn't need much assembler, to
do things like stack swapping. Control exchange is one of those things
completely missing from C.

------
damian2000
_If programmers wanted a statically-typed, compiled system language for their
scripting efforts, they would be using the existing system languages for that
already._

I think Go has been fairly successful at Google in terms of improved code
maintainability, and as an alternative to not just C but C++.

~~~
slurry
It seems to me that they are, in fact, using "statically-typed, compiled
system language[s] for their scripting efforts" - that is in fact what people
are doing with server-side Java, Scala and C#, not to mention Java/Dalvik and
Objective-C on mobile platforms.

Perhaps this article misses the point a bit. For several decades we have been
using scripting languages to glue together sharp pointy C[++] tools
(sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from
within (VB/elisp approach).

More recent practice, on both mobile and server platforms, has been to
abstract away from C[++] with a middle layer of memory-managed, compiled C-ish
goop - JVM, .Net, Objective-C, whatever. This is the level where people are
doing everything from amazing high-availability big data things to amazing
little bleep-bloop handheld things. I think the answer to "what's the state of
low-level systems programming?" is "not much is going on, but damn, there's a
lot of fresh stuff happening in mid-level systems programming."

Go and Rust are late entrants to this party, and probably exist more as a
hedge against Oracle litigating Java/JVM into the ground than anything else.

------
lispertoascheme
The state of programmers trying to avoid coding in the language their system
is coded in. C.

------
martincmartin
_For example, Java and C# cannot walk like a duck, not even if that is what it
would take to save themselves from drowning. Therefore, they are absolutely
unsuitable as system languages._

Hadoop held the world record for sorting a Terrabyte for a while. If sorting
large amounts of data as fast as possible isn't a system's programming task,
then I'm a three-headed monkey.

------
pretoriusB
> _The Go language and the Rust language. What they have in common, though, is
> that both Go and Rust are entirely incapable of walking like a duck. Since
> you cannot call Rust -or Go code from a scripting engine, they are
> inadvertently positioning themselves as alternatives to the scripting
> engines themselves._

Before writing all these BS, maybe he should do a little research first?

Here's Rust (which is not even released yet) being capable of being called
from C. It's a feature they are actively work on:

<https://github.com/mozilla/rust/issues/1732>

~~~
bluehex
> Before writing all these BS, maybe he should do a little research first?

There's a subtle difference between being able to call _back_ into Go/Rust
code from C (where the main function is in Go/Rust) which both languages allow
now, and allowing a call to originate from a C program, which neither
currently do. The author is talking about the later.

I would agree the article could go into more detail about what's currently
possible, and why it's not enough. But calling BS, I think, is going a bit too
far.

~~~
pretoriusB
> _and allowing a call to originate from a C program, which neither currently
> do._

That one is also under consideration by the Rust developers.

------
frujka
it's probably an interesting read except that being hosted on google's
blogspot means it only display blank pages due to content being embedded in
some javascript shenanigans.

Even a screenshot would be better than this.

Please move on to some actual html website that just works or rehost the
content elsewhere before posting it here. Thanks.

~~~
eropple
I used to be one of These Guys who never used JavaScript, too, but honestly?
It's 2012 and JavaScript is a thing pretty much everywhere worth targeting.
Demanding that somebody cater to your microsegment is a little much.

~~~
vq
Although I'm the kind of guy who usually have JS disabled I could be made to
see your point. But this page is actually a good example of why I run with JS
disabled. It magically positioned links and boxes over the scrolling bar and
caused my initial attempt at scrolling down to navigate away from the page.

Disabling JS usually have the effect of doing away with such nonsense but here
that is no option.

This time it was worth the hassle since it was an interesting article, but I
definitely prefer it when JS is optional.

