
Why I think Rust is the "language of the future" for systems programming - pcwalton
http://winningraceconditions.blogspot.com/2012/09/rust-0-index-and-conclusion.html
======
kibwen
Out of mild concern over the title (not that it ought to be changed, TFA
doesn't really have a meaningful title), I'd like to preemptively defuse any
potential flame war.

Go and Rust are not really competing. There may be some overlap in domain, but
they occupy different niches and will likely appeal to different crowds. Go
will appeal more to people who prefer its focus on conceptual simplicity and
its "opinionated" nature (very much like Python). Rust will appeal more to
people who prefer "functional" trimmings (algebraic datatypes, pattern
matching, liberal use of higher-order functions, Scheme-style macros) as well
as folks fed up with C++ who aren't willing to give up complete control over
memory and performance characteristics. The fact that both languages are built
around a similar concurrency model is just convergent evolution.

It's tempting to cast Go vs. Rust as Google vs. Mozilla and/or Chrome vs.
Firefox, but there's no practical reason that both languages cannot peacefully
coexist and thrive.

~~~
pcwalton
The two languages aren't really in the same space in the first place. Go is a
simpler language that leans more heavily on garbage collection. Rust is a more
complex language that can be safely used without the GC at all.

Go is a great language -- I greatly admire its simplicity -- and for its
domain it's fantastic. Rust is in a different domain: low-level systems
programming in which abstractions must be zero-cost and control over the
machine is crucial.

~~~
shanemhansen
Go actually provides what I believe to be zero cost abstractions that map
straight to the c memory model, making it very easy to use for systems
programming.

If you want a fixed size array of bytes, for for it. Zero overhead. If you
want a slightly smarter list you can use a go slice, which is also pretty low
overhead. I've personally verified that if I create a struct in go it takes up
the exact same amount of memory as the same c structure.

As a concrete example, I recently built an interface to kernel crypto apis
using pure go (no cgo). There's no magic to go data structures, you can define
structs in go and pass pointers directly to syscalls (through the syscall
package) and everything just works. Dealing with large arrays of binary data
is similarly straightforward. So go does give you complete control over memory
layout if you choose to use it.

The elephant in the room is that go does have garbage collection, and that
ain't free. Practically you can minimize it's impact by managing your own
memory. In fact that's what Brad Fitzpatrick is doing with his go based
memcached implementation.

It all boils down to how you define systems programming. I guess if you mean
is go suitable to write a kernel in, the answer is probably no, (but it would
sure be fun to try). If systems programming requires having the complete
ability to interact with the operating system and all syscalls + tight control
over memory layout, then maybe go is the way to go.

<https://github.com/shanemhansen/gocryptodev/>

[edit]

with respect to python and garbage collection, did you know you can actually
turn it off? If you have no cycles in your program you can turn off gc and let
the ref counter clean up all your data, similar to Objective-C's ARC.

<http://docs.python.org/library/gc.html>

~~~
pcwalton
There is a lot more to zero-cost abstraction than memory layout. If you do
manual memory management in Go, then you get no safety. It is also difficult
to use the standard library with manual memory management. Furthermore,
interfaces in Go are not zero-cost; you incur virtual dispatch whenever you
use them.

The article is about doing _safe_ manual memory management. That's something
that's unique to Rust.

------
haberman
I'm a die-hard C guy. My motto for years has been "you can pry pointers and
address spaces from my cold, dead hands."

Of the new languages I've seen lately, Rust is my favorite. I love how it
gives me better ways to express things I actually want to say without imposing
GC on me.

But even so, I can't see myself actually using it for much, because writing in
a language other than C means buying in to that language's runtime. Buying
into one language's runtime means that your code won't play nice with _other_
languages' runtimes.

If I write a library in Rust, how can I expose my types and algorithms to
Ruby, Python, Lua, etc? How will Rust Tasks play with Python threads? What if
I use a Rust Pipe to send a Python value between tasks? How do I keep Rust
from doing a GC pass while I'm holding the Python GIL? etc. etc.

Programming Languages by their nature want to be at the center of your world.
If you buy into their abstractions, everything works nicely. But if you try to
mash two of them together in a single process, you start to suffer from the
fact that their abstractions overlap and don't interoperate at all.

If you're only writing an application (ie. not a library) and never want to
embed other languages into your application, then this might be ok. But I'm
more interested in writing shared functionality that is useful _across_
languages. Why should the whole stack of parsers, crypto, compression, etc.
have to be written separately in each language? Life is too short to do some
great work that is only usable by one language community -- computing is so
big and changes so much that one-language-only functionality is at best
limiting your market and at worst dooming your code to obsolescence when the
next big language comes around.

So as much as I instinctively like Rust, I think I'll be sticking with C.

~~~
pcwalton
"If write a library in Rust, how can I expose my types and algorithms to Ruby,
Python, Lua, etc?"

The same way you expose them in C. Rust and C are compatible at the binary
level.

"How will Rust Tasks play with Python threads?"

More or less the same way C setcontext()/swapcontext() workalikes play with
Python threads. We probably want a runtime-less Rust to allow users who aren't
using tasks at all to just omit the whole system, though. The language itself
knows nothing about tasks; they're purely part of the runtime library.

"What if I use a Rust Pipe to send a Python value between tasks?"

Should work as you expect. In Servo we're already sending Objective-C values
(which require special APIs to perform the memory management) from task to
task over pipes.

"How do I keep Rust from doing a GC pass while I'm holding the Python GIL?"

By not using the GC. It's easy to tell when you aren't using the GC; you can
just avoid @ types, and there is a warning you can turn on to enforce no GC
use. This sort of thing is the reason why we support manual memory management.

"If you're only writing an application (ie. not a library) and never want to
embed other languages into your application, then this might be ok. But I'm
more interested in writing shared functionality that is useful across
languages."

We're already successfully embedding a JavaScript engine into a pure Rust
program for Servo. JavaScript can access Rust types and vice versa.

"Why should the whole stack of parsers, crypto, compression, etc. have to be
written separately in each language?"

I agree completely that this is undesirable. That's why Rust doesn't do this.
For your three examples, Rust and Servo link to C libraries: Hubbub for HTML
parsing, NSS for crypto, and zlib for compression.

~~~
haberman
Thanks a lot for the info!

> We probably want a runtime-less Rust to allow users who aren't using tasks
> at all to just omit the whole system, though.

I'd be really interested in reading more about this if/when it is available.
I'm also interested in whether you'd ever ship the tasks library standalone,
such that you can interoperate with Rust tasks from C (obviously you'd have to
follow certain rules to maintain the integrity/safety of the runtime).

> I agree completely that this is undesirable. That's why Rust doesn't do
> this. For your three examples, Rust and Servo link to C libraries: Hubbub
> for HTML parsing, NSS for crypto, and zlib for compression.

Understood, but I'm interested in the story for the guy who is _writing_ those
libraries.

~~~
radarsat1
I just wanted to say, I totally understand where you're coming from here. I
often end up writing libraries in C _explicitly_ because that's the easiest
way to hit as many targets as possible and not impose any unexpected runtimes
on other languages, even if I know it would be easier to write in another
language. So, I would say that not enough language designers think this way,
and it's really refreshing to see you describing this point of view.

I've often thought that a useful exercise would be to come up with a language
that adds just a bit more power than C but stays within the boundaries of the
C runtime, just for this purpose.

However, if new languages like Rust can deliver this, all the better! One
language that I once looked at with this in mind was Clay[1], which seemed to
basically be C with more safety and generics. However, it doesn't seem to be
as nicely supported as Rust, and perhaps Rust's static analysis and
concurrency support will be more powerful.

[1] <http://claylabs.com/clay/>

------
megaman821
Rust is what I had hoped Go would be.

Google employs some of the brightest computer science minds in the world and
turns out stuff like Go and Dart, which seem to be more aimed at enterprise
Java programmers rather than computer scientists or programming enthusiasts.

~~~
kibwen
Neither Mozilla nor Google are particularly keen to faff around designing
languages for the hell of it. :)

Google needed to ease the burden of hours-long Java/C++ compile times on
massive projects. Hence Go.

Mozilla needed a language that was as fast as C++, but safer and trivially
parallelizable. Hence Rust.

Beyond these goals, the fact that the rest of the world is excited for these
languages is just gravy.

~~~
neilk
You attribute too much to these entities called "Google" and "Mozilla".
They're made of people.

No executive asks for a new programming language. However at some enlightened
organizations, they are willing to let hackers explore radical approaches.

At some point, in order for them to become "official" projects, the hackers
have to align the language with the organization's goals. But the imprimatur
of their creators is unmistakable.

In any case, I doubt there is any plan to move everything in Google to Go.
They're willing to let the creators and some enthusiasts play around with it,
maybe deploy a few apps internally. But for Rust to be a success, Mozilla's
going to have to use it extensively in their main product.

~~~
InclinedPlane
Visual basic. C#. Powershell. F#. Javascript. Erlang. Fortran. Objective-C.

All languages developed intentionally by corporations.

Additionally, there have been languages developed by committees, such as
ALGOL, COBOL, CPL, Haskell, Ada, and many others.

~~~
neilk
Yeah, I admit my comment was not all that well thought out.

I was reacting to the usual dumbass theory that $BIG_ORGANIZATION is executing
some master plan which explains all of their actions. It explains how they
filter and rank ideas, but it doesn't explain why those particular ideas
happened to be floating around in their organization.

When it comes to things like new frameworks and languages, in my experience,
it's usually some employee going off on their own. And the reason why the
language is the way it is has more to do with that person's obsessions and
interests. They later justify it to the organization by producing results and
getting peer buy-in. FLOW-MATIC (ancestor of COBOL), Perl, Java, Sawzall, Go,
and Rust seem to be like this.

The other pattern is when companies intentionally make a language to compete,
or lock-in a developer platform. Now this move really _is_ predictable, and
any MBA can draw you a 2x2 matrix showing why Microsoft had to create C#.
Likewise VB, PowerShell, F#, and JavaScript. It doesn't mean they are bad
languages. But they are often platform specific since their whole purpose is
to corral developers into one camp.

I don't think Haskell counts at all. That was an academic project amalgamating
other academic projects.

I don't know enough about the others you mention. Maybe this isn't a useful
way to think about programming languages, but I was just exploring the idea by
rambling about it. ;)

------
james4k
Three different types of pointers? Perhaps that is going a bit far?

> If you've a sharp eye, you're wondering what that "~" is that I snuck in on
> the type of the closure for the child task. That's actually a pointer type,
> of which Rust has three (none of which can be null, by the way):

> ~T is a unique pointer to a T. It points to memory allocated in the send
> heap, which means data inside of unique pointers can be sent between tasks.
> You can copy unique pointers, but only by deeply copying (otherwise they
> wouldn't be unique!) (and by default, they are "non-implicitly-copyable", so
> the compiler will issue warnings if you copy them without writing the "copy"
> keyword).

> @T is a managed pointer to a T. Currently, these are reference-counted and
> cycle-collected (they may be full-on GCed in the future). Copying one
> increments the reference count, so multiple managed pointers can point to
> the same data. These are allocated on a per-task private heap, and cannot be
> sent between tasks.

> &T is a borrowed pointer to a T. It can point to the inside of arbitrary
> data structures - on the stack, inside ~ or @ pointers, etc. Rust has a
> static analysis, called the "borrow checker", that ensures that borrowed
> pointers must not outlive the scope of the pointed-to data (i.e., it is
> impossible for rust programs to have a use-after-free).

~~~
scott_s
Do you have reasons why it's going too far, or is that just an emotional
reaction? Consider that these pointer types map exactly to the pointer-type
templates provide by C++11: unique_ptr, shared_ptr and weak_ptr:
<http://en.cppreference.com/w/cpp/memory>

In the interest of starting discussion and not just stating facts, I will take
the position that I think Rust's adoption of these concepts into the language
is a Good Idea. Many pointer-based codes have these pointer types, but they
are enforced through convention alone. C++11's templates are a step in the
right direction, as they make explicit what convention assumes, but the
compiler does not know about them. Yes, because they are template classes,
there are certain things one can and cannot do with, say, a unique_ptr, but
the compiler cannot do any _analysis_ on the usage, because the concept has no
representation in the language semantics. (Ben references a blog post by Niko
Matsakis about some of the analysis Rust does:
[http://smallcultfollowing.com/babysteps/blog/2012/07/19/yet-...](http://smallcultfollowing.com/babysteps/blog/2012/07/19/yet-
another-tutorial-on-borrowed-pointers/))

~~~
cmccabe
A lot of the productivity gain in higher-level languages comes from not having
to manage memory. Having three different incompatible, differently allocated
pointer types to fool around with slows you down. The creators of Rust are
hoping that the speedup from not having to do global garbage collection in
most scenarios will balance out the loss of productivity.

Personally, I'm skeptical. Azul showed us that pauseless GC was possible, even
for Java. Android and .NET showed us that even without pauseless GC, the
performance of GC'ed languages was adequate to build attractive user
interfaecs. A lot of Rust's design choices mean that it's not really suitable
as a web development or scripting language, and those two communities have
shown themselves the most receptive to new languages.

Mozilla has tried to rewrite their core product in a different language
before. Look up "Javagator." It did not end well. But, who knows. Maybe this
time really will be different.

~~~
pcwalton
It's not _impossible_ to build attractive user interfaces with garbage
collection, but it is significantly harder, both for users and developers of
the system. Android system apps go to great effort to avoid using the GC
during critical animations, while in iOS this is much easier. Implementing a
good garbage collector is a _lot_ of work, and it's very difficult to tune;
even the JVM, which has had an enormous amount of man-hours put into it,
struggles with GC pressure under some loads.

Additionally, the applications we want Rust to be suitable for aren't
competing with GC'd languages. They are competing against applications written
in C++, with manual memory management and no automatic storage reclamation.
They will be put head-to-head in benchmarks with them. We don't take
performance regressions, and relying on global GC would be a significant risk.
The Javagator is, in fact, an excellent example here!

(As an aside, the pointer types are not all incompatible; all pointers can
become &T, and the vast majority of functions take & pointers.)

~~~
cmccabe
It's a little misleading to say that GC makes "attractive user interfaces...
significantly harder." For any business application, for example, using GC is
a no-brainer. That's why I mentioned .NET in my response.

Android games have been known to struggle with GC sometimes. However, garbage
collection has gotten a LOT better on Android over the years. I wrote a small
game in Android 1.1, back when GC pauses were anywhere from 100ms-250ms, and
avoiding allocations was VERY important. Nowadays, most pauses are less than 5
ms, and the NDK is always available if you want to write native code and avoid
GC altogether.

In general, the game industry seems to be moving towards a model where you
have "engines" and "level packs." The physics, graphics, etc. engines are
developed by a small group of people in C or C++, and the level packs are in
whatever scripty language you want.

The development of the web reflects a similar split. You have the C/C++
engine, and the HTML/CSS/ECMAScript content. C and C++ have low programmer
productivity and a lot of odd quirks, but nobody cares because the bulk of the
development has moved higher up the stack. So attempts to replace C or C++ get
a "if it's not broke, don't fix it" type response.

This is one reason why golang, for example, has seen more users come in from
the scripting and web development community than from the C++ community. C++
projects tend to be old, conservative projects that are at the bottom of some
giant stack-- like Google's infrastructure, for example.

Rust has some interesting ideas. The typestate concept in particular is
interesting. I also do kind of get what they're going for with the 3 different
memory types. I guess we'll see how it all works out.

------
jallmann
I see Rust has ADTs and strong static typing, how much is enforced at compile
time?

One thing I love about OCaml is its exhaustiveness checking. I've been doing a
lot of Erlang and it feels so dangerous without the compiler telling me if
I've missed a pattern, or if a clause might return the wrong type. (Dialyzer
and typespecs help somewhat, but aren't nearly as nice.)

If I could have that sense of safety with Rust, I would be persuaded to
investigate it sooner rather than later.

~~~
bblum
Short answer is "all of it"[1]. 'match' statements in rust also exhaustive. I
believe ADTs are used and checked exactly the same way as in ocaml (I've used
SML and Haskell and they are the same as there).

[1] not to say dynamic failure is totally gone from the language -- but it is
if you avoid both writing 'fail' and using library functions that can fail. A
common example is option::get(Option<T>) -> T, which fails if the optional
value is None -- it leaves a sour taste in my mouth whenever I use it, but I
still do sometimes.

------
Calamitous
Has anybody heard if they've finalized a spec yet, or at least come close?
I've thought about getting into Rust from time to time, but the lack of a
finalized spec and/or stable reference implementation has always scared me
off.

~~~
pcwalton
We're getting close. Rust 0.4 (hopefully to be released next week, assuming no
more schedule slips) will be quite close to final syntax-wise. Although not
all the features will be implemented, your code will be mostly backwards-
compatible.

If you'd like to try out the language, I recommend working off git master
rather than 0.3; the language has been dramatically simplified in Rust 0.4.

~~~
Calamitous
Excellent, thanks for the update! :)

------
erichocean
_which is especially important in the multicore world where heaps must be
protected by a global lock_

The article seems nice, but this is misleading at best, and flat out false at
worst.

Concurrent memory allocation has been around for many years, and _gasp_ is
perfectly usable with C.

~~~
bblum
I wondered how long it would take for somebody to call me out for that. It
seemed like a sketchy claim when I wrote it (it wouldn't be too complicated to
make the @-heap lock-free in rust, for starters), but in retrospect, I
shouldn't have said it at all. Thanks for pointing that out.

~~~
bblum
I tried to think of an alternate justification, but couldn't come up with a
non-awkward way of saying it, so just redacted that part entirely. I'll say
here, though, a few reasons I think avoiding malloc/GC is important:

\- pcwalton has mentioned his desire to be able to reason about when the
garbage collector runs, so that he can statically guarantee that the graphics
rendering task in servo will never take a big pause to do a GC.

\- In this-must-never-fail environments, such as writing kernel device
drivers, using dynamic allocation has to be avoided (and, if i'm not mistaken,
even more likely to fail when done in an "atomic section" using
kmalloc(GFP_ATOMIC)). Rust in its current state is not suitable for kernel-
level development, but it's not far off; and it's critical to be able to
reason about memory allocation for such applications.

------
BrandonM
From the end of the last post in the series
([http://winningraceconditions.blogspot.com/2012/09/rust-4-typ...](http://winningraceconditions.blogspot.com/2012/09/rust-4-typesafe-
shared-mutable-state.html)):

    
    
        let arc = RWARC(some_data);
        for num_cpus().times {
            let arc2 = clone(&arc);
            do task::spawn |move arc2| {
                process_data(arc2); // might read, write, whatever
            }
        }
        let modified_data = unwrap(move arc); // blocks on all child tasks at once
        // do more of the algorithm, etc.
    

This is a nice fork/join pattern, allowing implementations with minimal
copying and reasonable safety. If this is indicative of design decisions made
by the creators of Rust, I think it looks pretty good.

------
zobzu
I don't _know_ if it _is_ the language of the future, but Rust is definitely
high on my "let's reprogram Singularity/Plan9/Friends"-and-actually-use-it
list

I'd certainly join and attempt to help any project started in that direction,
at least.

~~~
eholk
It's interesting that you mention Singularity, as it turns out that
Singularity has been a major influence on Rust. For example, the current
communication is very similar to the channel contracts used in Singularity.

~~~
zobzu
Well I'm a big fan of many concepts of Singularity. And I like rust. I guess
that's no coincidence ;-)

------
mappu
How do you implement closures entirely on the stack? What about the case when
a function returns a closure into an external translation unit? How do you
cover the bound parameters? You can't do call-time lambda lifting if the bound
parameters are out of scope at the call site, you need to put something on the
heap and GC it. Same with compile-time lambda lifting (producing a chain of
bind1st stubs on the executable heap)

~~~
kibwen
I'm not an expert, but I believe that Rust will allocate closures depending on
the data that they close over. So if you reference a stack-allocated variable
from within your closure, it gets put on the stack. If you reference a heap-
allocated variable from within your closure, it gets put on the heap.

I could be dreadfully wrong, though.

~~~
pcwalton
No, where a closure is allocated depends on the type of the closure. The type
of the closure is inferred just like any other type. It has nothing to do with
what it closes over (although what it the closure is _allowed_ to close over
depends on its type).

------
philhippus
>Stack-allocated data lets you often avoid dynamic allocation overhead and
garbage collection (even closures can sometimes be entirely on the stack).

There's an entire website named after the results of over-using the stack.
Smart pointers and move semantics in C++11 completely smoke GC and memory
management everywhere else.

While I'm sure Rust is a nice language and I applaud its innovation,there's
already a perfectly good wheel.

~~~
argv_empty
_Smart pointers and move semantics in C++11 completely smoke GC and memory
management everywhere else._

Aside from having to enforce proper pointer usage at runtime, whereas Rust
does it at compile time.

------
adambratt
Good article!

Sorry to be nitpicky but the font size/line-height makes it really hard to
read. Perhaps make both a bit bigger for optimal reading?

~~~
kibwen
I just found this Firefox add-on today and I'm completely in love:

<https://addons.mozilla.org/en-US/firefox/addon/clearly/>

Here's what appears to be the Chrome version:

[https://chrome.google.com/webstore/detail/iooicodkiihhpojmme...](https://chrome.google.com/webstore/detail/iooicodkiihhpojmmeghjclgihfjdjhj)

Makes it ridiculously easier to read basically everything on the web.

~~~
cpeterso
I like the Readability add-on and bookmarklet:

<http://www.readability.com/apps>

------
ww520
Rust looks very interesting. I downloaded it and tried it out on Windows but
got "libgcc dll missing" error. They really need to statically link in the
MingGW runtime to make it easier to run.

~~~
pcwalton
There is a bug on doing that, but keep in mind that bundling libgcc would only
make Rust error out with a more informative error. Because Rust uses the
system linker, you need MinGW installed to use Rust.

------
marshray
Someone ought to mention the similarities between Rust and Scala. These are
two really great languages, IMHO.

------
shmerl
Rust indeed looks very promising. I hope it'll pick up usage beyond Mozilla
internal projects.

------
drivebyacct2
Rust and Go provide me with more day-to-day pleasure than... well... a sadly
large number of things. This is a good resource for learning more Rust,
thanks.

------
elchief
why didn't they call it YAPL?

------
EternalFury
Another language that adopts a different syntax for the sake of adopting a
different syntax. If you are going to ask me to adopt a different syntax,
you'd better have a good reason for it, because I can do everything I need to
do with the languages I already have.

~~~
bryanlarsen
You can do more in assembly language than you can in any other language. That
doesn't make it a good language to program in except in very special cases.

