
Rewrite Everything in Rust - fabrice_d
http://robert.ocallahan.org/2016/02/rewrite-everything-in-rust.html
======
panic
I think this misses the real problem. So many pieces of foundational software
like glibc and OpenSSL are understaffed, underfunded, and plagued by terrible
code. Go read glibc getaddrinfo: it's a mess!

Rewriting the software in Rust would not solve these problems any more than
rewriting it in C++ would. A rewrite would clean up the code, sure, but then
you're left in the same situation, only with brand new bugs that nobody has
time to fix.

We need to have incentives for maintaining this foundational software.

~~~
pcwalton
> A rewrite would clean up the code, sure, but then you're left in the same
> situation, only with brand new bugs that nobody has time to fix.

The entire point is that you're _not_ in the same situation regarding memory
safety problems/vulnerabilities.

~~~
rubiquity
I think what GP is saying that you'll have problems in logic or other typical
programming problems and that it isn't a lack of language features/safety but
rather time/money being thrown at these libraries

I imagine the top two reasons unsafe memory access happen is:

1) other complicated logic seeped into the memory sensitive area or causes
programmer fatigue

2) memory management is hard

Rust helps with #2 but let's not kid ourselves, what percentage of a library
like glibc would be spent in unsafe blocks? Drawing attention to an unsafe
area can help though.

As you can see, I'm running around in circles. Just like every discussion
about this does. Until an avid Rust user puts their money where their mouth
is, we're all just playing Armchair Programmer.

~~~
bluejekyll
> an avid Rust user puts their money where their mouth is

There are lots of people doing this. Look at crates.io. I decided to write a
new fully compliant DNS server and client, trust-dns. I work on it in my spare
time (it's time not money), which is hard with two kids.

I've learned a few things. It's hard to get it done. There's a lot to learn
from the rfc's. Just because it's Rust, doesn't mean it's easy. There are all
the same borrow rules, and I know I have a bunch of TODOs in the code to go
fix things like extra clones. The IDEs, while good, are not on par with other
languages yet.

Basically it's a lot of work. And I commend the individuals who've paved the
way for me with the std library in Rust, but we still have so much more work
to do until there is a full OS env like GNU.

~~~
rubiquity
Your entire post reinforces the idea that in my list of challenges to writing
safe software that #1 is far more the issue than #2.

~~~
bluejekyll
Yes. I'm agreeing that it's hard and takes a lot of energy. But people are
doing it.

------
carussell
Do user studies.

It's pretty much unheard of, but nearly every open source project would
benefit far more from a week of watching potential contributors trying to get
up to speed than it would from making sure the project roadmap is delivered a
week sooner.

That's my claim for projects in general, and what you're asking for re Rust is
a little different, but the approach can applied there, too. Focus on
everything from navigating the project website and other docs, to interactions
with the tools that make up the "ecosystem", to programmer expectations that
could shape the language (with the latter actually being the least important).

Don't trust users to self-report. Don't overestimate your ability to make a
combined diagnosis and prescription, and don't _under_ estimate how much
silent resignation is killing your adoption rate. Observe and work off real
notes.

~~~
charlieflowers
This is the most insightful comment in the discussion. It should have been
obvious, but it wasn't until you posted it.

First, design a sound language that solves problems other languages don't
(done). Then, optimize for adoption ... because the world only benefits from
that new language once adoption really takes off.

Rust has done half the job really well. And a decent start on the second half
-- mainly because the first half was done so well.

But if the Rust team were to prioritze adoption the way the parent post
espouses, they'd be changing their priorities significantly, and it's probably
the most effective way to make sure Rust fulfills its potential.

~~~
steveklabnik
We have done a lot of things to prioritize adoption, though we could do
better.

We have done some almost-studies like this: we had a series of conference
calls with companies using production Rust specifically to work on their pain
points, for example.

------
gnu
Rust is nice. But the article miss one important point. Modern software
engineering is a collaborative effort (well, it has always been, afaik). I
write code for others to understand. I find Rust code significantly hard to
read (I am saying this as an experienced C/C++ programmer and as an
intermediate haskeller). Yes, one can write write-only code in any language. I
think it is extremely important to write code that others can understand and
imho, it trumps all other reasons. Not trying to underplay the value of type
safety. It is fantastic. My Haskell programs have much much less bugs and I
tend to get it to work, much faster than when I do C/C++ code. I would expect
some of that goodness to be in Rust programming as well.

It would have been fantastic, if Rust took only a few steps further from
Cyclone. I also wish that Rust authors stop adding more and more features. The
target audience, namely low level system programmers, tend to work close to
hardware and think along those lines, so bombarding them with type theory
concepts is only going to shoo them away.

Just my 2 cents.

~~~
kibwen
And as a Rust and Python programmer, I find C++ hard to read (I can never
shake the feeling that the ampersands are always in the wrong place), and I'm
nearly hopeless at deciphering Haskell. :P I suppose this is just something
that comes with experience.

~~~
kyllo
Haskell syntax is really nowhere near as complex as C++'s or even Rust's. It's
not indecipherable, it's just different and not C-like. It's in the same
family as SML, OCaml and F# so if you learn one of those, the others come
easily.

~~~
FreeFull
I suspect Haskell's syntax isn't as much of an issue as the typical style that
Haskell code tends to be written in; really short, abbreviated variable names,
and very high code density.

------
dkhenry
I have actually started to do this to see for myself how much work it would be

[http://blog.dkhenry.com/2016/02/17/deciding-to-rewrite-
getad...](http://blog.dkhenry.com/2016/02/17/deciding-to-rewrite-getaddrinfo-
in-rust/)

~~~
rtpg
Thanks for doing this...

I think the way to tackle a rust glibc rewrite:

Step 0 - write a code generator that just generates rust code that call the C
versions of glibc symbols ( based off of the publicly exposed API in the
library files). Get test suite to run

Step 1 - Make that rust code generate a 100% drop-in replacement fro the C
symbols (maybe this requires injecting version numbers)

Step 2 - write the `getaddrinfo` replacement in Rust

???

Step 4 - have all of glibc ported

EDIT: Does anyone know of any gotchas with linking that would cause this to
happen? Seriously considering taking a day off to do step O.

~~~
JoshTriplett
For step 0, look at Rust's libc crate.

For step 1, you'd need to carefully look at glibc's ABI, which is even more
complicated than its API due to backward compatibility and symbol versioning.

~~~
dkhenry
Yeah ABI compatibility might be problematic, my currently plan is to
LD_PRELOAD what I have to test it for speed. I have the feeling that rust
could mock the ABI and serve as a general replacement, but first things first
is it even fast enough to use at the lowest levels of the system.

~~~
steveklabnik
If it's not, please file bugs. It should be.

------
_b
C++ is getting compile time checks for type, bounds, and lifetime safety,
which steals nearly all of Rust's safety thunder.

See Herb Sutter's talk at CppCon: [http://herbsutter.com/2015/09/27/my-talk-
at-cppcon/](http://herbsutter.com/2015/09/27/my-talk-at-cppcon/)

For most applications, moving to modern C++ is going to be a better option
than rewriting in Rust.

~~~
ubernostrum
_For most applications, moving to modern C++ is going to be a better option
than rewriting in Rust._

I wouldn't bet on that.

Companies which go with a language like C++ don't do it for "safety" or
"performance" or reasons like that (at least, most of the time they don't).
They go with it because they can get something written once, then run it for
the next half-century and only have to spend engineering time on new features
and the occasional critical bug. Telling them they need an intrusive rewrite
of their entire codebase to use modern practices will simply result in either
(A) no action taken (most likely) or (B) the rewrite happening in a language
that isn't C++.

This is incidentally what happened with quite a few significant Python
codebases in the 2/3 transition -- rather than rewrite their code to Python 3
standards, they said "eh, we're rewriting anyway, let's go to Lua" (most
often, though some other languages were the "let's go to" option as well).

~~~
make3
I don't know that people don't choose C++ for performance reasons or for easy
manipulation of low level stuff. Games, high performance computing, embedded
and legacy software are the main applications of C++, aren't they?

Some people on this website seem to blow the python 2/3 transition fairly out
of proporotions.. Python 3 is really extremely similar to Python 2, and there
are great tools to convert Python 2 code to Python 3 and vice versa.

~~~
rspeer
Is there a great tool to convert Python 3 to Python 2?

I've tried Pasteurize (part of python-future) but it just leaves the "yield
from" statements in place, giving syntax errors. And 3to2 does even less, and
seems to mostly be a framework for building tools like Pasteurize.

I know this isn't easy, but JavaScript people do it with ES6 transpilers, and
ES6 is more different from ES5 than Python 3 is from Python 2.

~~~
JelteF
The two libraries I've created both use a combination of 3to2 and pasteurize
to convert the python 3 code to python 2. The only issues I've run into are
new error types that I could not use. But there seem to be quite some issues
open:
[https://bitbucket.org/amentajo/lib3to2/issues?status=new&sta...](https://bitbucket.org/amentajo/lib3to2/issues?status=new&status=open)

In case you want to see the scripts I use to change the python3 code to python
2 and make it work with pip:

[https://github.com/JelteF/PyLaTeX/blob/master/convert_to_py2...](https://github.com/JelteF/PyLaTeX/blob/master/convert_to_py2.sh)

[https://github.com/JelteF/PyLaTeX/blob/master/setup.py](https://github.com/JelteF/PyLaTeX/blob/master/setup.py)

------
daurnimator
Rust has major unsafeness around low memory that make it unsuitable in this
libc type role. See:
[https://www.reddit.com/r/rust/comments/2mthq2/how_would_a_ru...](https://www.reddit.com/r/rust/comments/2mthq2/how_would_a_rust_application_be_able_to_properly/)
and [https://lwn.net/Articles/644708/](https://lwn.net/Articles/644708/)

Due to this issue that the rust authors seem unwilling to address, I fully
recommend _against_ rust for the precise roles it's intended to be good at.

~~~
hermanradtke
Based on my reading, only the std library has the behavior unsuitable for a
libc type role. I would think a libc replacement would use #[no std] and have
the finer grained control.

~~~
roca
Exactly.

------
geofft
It's certainly possible to incrementally rewrite an OS distribution in Rust,
or even smaller projects like glibc.

Anyone want to set up the infrastructure so it's easy to take a library or a
.c file and start hacking, and see if you still get a working OS in the end? I
suppose part of the problem is defining "working OS" \-- I'm not sure I know
of any great test suites that check to see that a GNU/Linux distribution
continues to in GNUy/Linuxish ways, which is usually what you want for a
large-scale rewrite project.

~~~
cls59
How about a re-write of coreutils in Rust?

[https://github.com/uutils/coreutils](https://github.com/uutils/coreutils)

------
pjmlp
How I wish people understood this when we were comparing Ada, Modula-2,
Modula-2+, Modula-3, Pascal dialects, Oberon, Oberon-2, Active Oberon,
Component Pascal,... to C.

Apparently we were the fools.

But this will never happen in UNIX OSes.

The culture is married to C, regardless how many memory corruption exploits
per days might get a CVE entry.

However I hope to be proven wrong.

~~~
dpc_pw
The syntax is killing these languages. Sure it's superficial, but programmers
are human beings and I doubt most people find it appealing to voluntary stare
at and type PROCEDURE BEGIN END and other LONGCAPITALIZEDWORDS...

~~~
pjmlp
Most programmer friendly editors for those languages supported formatting as
you type, gofmt isn't the first of its kind.

Also it makes very easy to recognize reserved keywords.

~~~
dpc_pw
BEGIN COMMENT No matter what editor does, programmer still has to look at this
ugliness. :D END

~~~
pjmlp
Beauty is in the eyes of the beholder. :)

------
thinkpad20
I don't know much about glibc, but since you can call rust from C and vice
versa, it seems like you could reimplement a few functions at a time,
incrementally phasing out C code and replacing it with rust. Assuming, of
course, that you agree with the author's thesis. This approach lets you reuse
the same tests and do things like benchmarking etc to make sure the
implementation is performant and compliant.

~~~
kibwen
This is exactly how Firefox is approaching it. Identify the most security-
critical bits, wall them off into their own component, and rewrite it in Rust.

~~~
bjz_
Yes! And the reverse has happened with Servo. Originally it was mainly a bag
of C libraries with some layout code, then gradually they began swapping the
pieces out for the Rust counterparts. This is how new software can be written
- bootstrapping off the C infrastructure, then gradually transitioning more
and more into Rust land.

------
mahyarm
How practical is rust for small processor IoT applications? Like the ESP8266
we saw recently?

[https://news.ycombinator.com/item?id=11148129](https://news.ycombinator.com/item?id=11148129)

They often have horrible security, but are very small programs when you use
them as wifi temperature sensors or similar. They are good targets for
rewrites and greenfield applications. Right now it seems like ardunio C++ or
some scripting language is the target.

~~~
pcwalton
Rust is a great choice in theory for anything with an LLVM target, but the
libcore "bare metal"/"no standard library" ecosystem is pretty immature, and
there are likely to be bugs (like the compiler generating binaries that are
too large, etc). If you'd like, please try it out and file any issues you find
:)

------
progman
Don't waste your time rewriting almost unmaintainable legacy stuff. Perl
actually died because the Perl 6 developers made the mistake to retain
compatibility with Perl 5 which was not possible due to its unmaintainability.
At first it looked promising but later on it turned out to be impossible. Rust
could fall into the same trap by retaining dependency on C/C++.

I would use Rust for a complete startover. Open source development should
shift its mentality to support open hardware. This could ensure safety of both
hardware and software in the future, and it would attract many developers who
want to contribute.

We already have the tools for open hardware: free IP cores, 3D printers, free
CAD, equipment for DIY PCB/SMD boards, etc. ... and Rust for bare bone
programming.

------
jeffdavis
"[Other safe languages] require complex runtime support that doesn't fit in
certain contexts (e.g. kernels)."

Complex runtimes are also unsuitable for libraries. If you write a great
openssl replacement in Haskell, it's only useful for Haskell applications.
Because nobody wants to link the Haskell runtime into many of the applications
that use openssl.

For widespread libraries, the choices are basically C, C++, and now
(hopefully) rust.

~~~
whyever
You can provide C bindings for you Rust library.

~~~
masklinn
Which is "for widespread libraries, the choices are basically C, C++, and _now
(hopefully) rust_.": as in C++ you can use high-level constructs internally
without relying on an extensive runtime, and expose and easy-to-FFI C
interface.

Rust still has insufficiencies though, the inability to customise the
(library's) allocator is one I think.

~~~
steveklabnik
You can use a custom allocator on nightly.

~~~
masklinn
Does the custom allocator support allow for runtime customisation e.g. the
library user calling it to set allocator functions then starting to use it?
And I expect the custom allocator is global?

~~~
steveklabnik
Currently, it replaces the global allocator. Using multiple allocators is an
interface that's still being actively worked on, it's in the RFC stage, so no
code has landed yet.

------
cpeterso
So how soon will we see a new libc writen in Rust? Someone could fork a
smaller libc like musl and incrementally rewrite functions in Rust.

------
_greim_
There's a perverse incentive in software, where toolsets requiring
memorization of arcana not inherent to the problem domain but inherent to _how
the tool maps onto the problem domain_ , will generate irrational loyalty
among its users. Our toolsets and the investments we make in them really do
become a way of life, and fuck the world if it tries to get us to change.

------
bitL
1: So a new generation, new rewrite of everything... Then hitting brand new
problems (sometimes old ones avoided by previous designs) not foreseen by
majority. A new language spawns with the coming of a new generation,
addressing some of these problems. Rewrites everything. Hits another set of
problems. Goto 1.

~~~
paulddraper
Iteration is not bad.

Iteration without progress is bad.

Which one do you think would happen?

~~~
bitL
I am suggesting that the problem is elsewhere - and more philosophical. I
understand for many very young kids learning JavaScript was the only
accessible way to do programming. Yet they are producing way worse stuff than
previous generations unfortunately (remember Scandinavian demos in the 90s?),
as their platform of choice is seriously limited. Now the article mentions
horrible things in code developed over decades. I can guarantee you most of
decade old code looks similar. And most likely decades old code in Rust will
be similar. Here is a need for some simplifying meta-programming theory
instead of slightly modified language constructs known since 60s.

So what I see is iteration with some progress and some regress (wrt to
empowering programmers and simplicity). It's good to have a new generation
enthusiastic creating their own world, yet if they are as human as previous
generation, most likely they would hit the same problems. Like a genetic
algorithm, a slight mutation to current solution runs for a certain time and
we see how the generation fares. But maybe we need completely different
approach to get out of local optima technology seems to be right now. Quantum
Rust? ;-)

~~~
paulddraper
"But maybe we need completely different approach to get out of local optima
technology seems to be right now."

So your problem is not that a Rust rewrite would be too much of a change. It's
that it would be _too little_? ;)

If there is any popular low-level language that is a "completely different
approach", I'd say that language is Rust.

~~~
bitL
Yes, too little ;-)

Frankly, Rust just takes a few different approaches known since "forever" and
promotes them to 1st class semantic citizens. Similarly Go takes another set.
Like every language in existence. So some things will be super easily
expressible in one or another language, other things more obfuscated. Yet you
could do the same in older ones, but perhaps more verbosely, with more
syntactic sugar or boilerplate structures. Hence the comparison to a genetic
algorithm with a few mutations in each generation. All a variation of the same
set of mental constructs, possibly with very similar descriptive abilities
(even when leaving Turing completeness out of the equation).

------
na85
Is there a "glibrust" or equivalent standard library for Rust, yet?

~~~
thinkpad20
Rust of course has a standard library[0] and it includes a libc, but (I might
be wrong here) I think the libc portion is FFI into glibc or similar. I think
the stuff written in rust is considerably more high-level, focusing more on
data structures, concurrency, string manipulation, etc.

[0]: [https://doc.rust-lang.org/std/](https://doc.rust-lang.org/std/)

~~~
steveklabnik
We use glibc by default but provide MUSL builds for some platforms as well.

------
wsha
How dependent is Rust on Mozilla for support now? Has it developed its own
community of contributors? I like Mozilla and their mission, but with the way
things have been going the past few years I wouldn't want to put a lot of time
into porting my projects to something whose future was dependent on Mozilla.

~~~
steveklabnik
As of just now, we've had 1323 contributors to the main repository, not
counting Cargo, some crates, or the website, for example.

Mozilla is paying virtually all of the people who are working on it full-time,
but there are a lot of people doing a lot of work in their spare time too.

    
    
      > with the way things have been going the past few years
    

If you're referring to the projects which have been dropped over the past few
years, I can see that. At the same time, Firefox is the core product, and Rust
is already in Firefox, so dropping it doesn't seem likely. The amount will
only be increasing.

------
stcredzero
If a given piece of software has been largely written using certain patterns
or conventions, then it is often possible for a syntactic rewrite engine to be
able to match those patterns, then output idiomatic code patterns in the
target language.

This may well be applicable to kernels and other constituents of an OS.

~~~
pcwalton
Isn't that just what a compiler is, where the target language is machine code?

~~~
wspeirs
I realize something like glibc would be the LAST set of source code to run
through a transpiler, but how hard do folks think it would be to make a C ->
Rust transpiler? I'd think you'd go about it by making Rust a target backend
for LLVM. This way you let LLVM deal with the all C stuff (macros, etc), and
just focus on the internal representation to Rust conversion. In theory, you'd
get C++ to Rust as well.

Chances is successfully going from some not-so-obfuscated C code to equally
not-so-obfuscated Rust code?

~~~
pcwalton
> how hard do folks think it would be to make a C -> Rust transpiler?

Why do you want to do that? The only way to make it work short of doing years
and years of static analysis research to automatically translate ad-hoc memory
management disciplines into those of Rust (that I think will not succeed
anyway) would be to compile to unsafe code. And if you compiled to unsafe Rust
code, there would be no gain.

~~~
wspeirs
I'm not sure it would take "years and years" of static analysis research. You
could use the data-flow analysis that is built into LLVM, then punt on the
rest. It doesn't HAVE to produce Rust code that compiles 100% of the time. The
goal would be to run it through the transpiler so you have a starting place to
produce Rust code that compiles and works the same as the original C code.

Again, this might be a terrible idea, but I'd be curious to hear from anyone
that has experience with this sort of porting...

~~~
pcwalton
Data flow has nothing to do with the hard problems you'd have to solve.

------
halayli
Then introduce a new set of bugs that will take years to iron out.

------
mback00
IMO -- Better is to just write better tools for plain olde ANSI-C. We can
build tools to check C as well as any other. Let's just not waste time on
building further complexity into our toolset and look at building toolkits and
best practice at the very flexible level of abstraction that Plain olde C
provides. With the right tools and expertise, we can improve the world without
the never ending whiz bang rewrite.

~~~
steveklabnik

      > We can build tools to check C as well as any other.
    

This is simply not true; language semantics determine the kinds of tooling you
can build. Ruby is so dynamic that it's notoriously hard to build tools for,
for example.

------
madebysquares
Sometimes I don't even feel the need to read the articles because the comments
are so good. Good stuff guys.

------
throwawyr424415
Doesn't modern C++, with smart pointers, auto, and the rest, cover most of the
same ground rust does, and isn't the Rust community somewhat toxic, with
activists pushing "Codes of Conduct" and other non-technical agendas?

~~~
roca
With modern C++ you can still easily shoot yourself in the foot and trigger
memory-unsafe behaviour.

Whatever your opinion of Codes of Conduct, lots of projects have them now. And
IMHO _not_ having a Code of Conduct encourages its own sort of toxicity.

------
alexnewman
I wish

~~~
alexnewman
Why -4 downvote?

------
latenightcoding
Not this crap again. C libraries are not pretty but they have been out there
for decades, they have been reviewed and used in production.

There is no silver bullet, just because you use Rust it doesn't mean your
programs will be completely safe.

A lot of these C libraries were written in more innocent times where a small
bug would not affect as many people as it would today.

We live in a C world and I don't see that changing any time soon.

~~~
Avshalom
>We live in a C world and I don't see that changing any time soon.

Of course it won't because every time an out-of-bounds array access gives
attackers free reign of a system and some one says "hey guys this whole C
thing is clearly blatantly terrible" every one else says "Not this crap again.
C libraries are not pretty but they have been out there for decades, they have
been reviewed and used in production."

As if the libraries have been utterly static for decades and aren't infact
full of holes and bugs, which is exactly what brings on the call to rewrite
things in Ada/Rust/SML/Lisp

Then they always say "There is no silver bullet, just because you use Rust it
doesn't mean your programs will be completely safe." As if there's no reason
trying to improve things unless you can jump directly to perfect.

they make excuses like "A lot of these C libraries were written in more
innocent times where a small bug would not affect as many people as it would
today." as if we still lived in those innocent times.

And then they inevitably wave the call to action off by appealing to the
status quo with some line like "We live in a C world and I don't see that
changing any time soon." and so even though there's no reason we couldn't
start re-writing things (especially in Unix an OS that constantly brags about
being built from loosely coupled parts) in a better language we never get a
critical mass together.

~~~
latenightcoding
I see what you did there

------
wangchow
Just gonna throw this out there: what if instead of working on creating the
myriad of "better languages" we focus on actual mastery of simpler languages
like C.

And with that mastery: what about writing some machine-learning software that
detects vulnerabilities as part of the build/release cycle. For instance:

[http://news.mit.edu/2016/faster-automatic-bug-repair-code-
er...](http://news.mit.edu/2016/faster-automatic-bug-repair-code-errors-0129)

To detect vulnerabilities people need to be careful and follow best practices.
If they can't do that well.. replace the people with software that's better
and more efficient than the people. :)

~~~
ubernostrum
_we focus on actual mastery of simpler languages like C_

We've had 44 years for people to master the "simplicity" of C. Turns out,
writing C is simple but writing safe, reliable C borders on the impossible
even for the most masterful masters who ever mastered their mastery of the
language. Better return on investment would be to work on killing C with
something that doesn't have C's problems.

------
sdegutis
Off topic, but it's surprisingly refreshing to see someone put "Christian"
next to their name in the same place they put "Mozilla hacker." The vast
majority of programmers either aren't religious or hide it so well that you
couldn't possibly tell if they were. (Not that being overt about it is better,
either.) I understand that it's generally best not to talk religion in the
professional workplace, but a person's own blog is a more neutral place to
bring it up, so it's nice to see that here. It's also refreshing to see that
there are other adult converts _to_ Christianity in the programming world, as
opposed to _from_ it, which seems to be the norm in general but particularly
in this field.

~~~
solipsism
Do you find it refreshing because he's Christian? What I mean is, if instead
it said "atheist" or "Muslim" would you also find it refreshing?

~~~
sdegutis
Being a Christian myself, yes, I do find it refreshing specifically because
he's _another_ Christian programmer. It's very rare that someone is both, let
alone an adult convert to Christianity.

