
Re: Integrating "safe" languages into OpenBSD? (2017) - xnyt
https://marc.info/?l=openbsd-misc&m=151233345723889&w=2
======
staticassertion
This is less "On Rust" and more "On accepting a rewrite of any tool into
OpenBSD, on the merits of memory safety alone".

There isn't really much of a statement or judgment on Rust. At most there's an
interesting point on it's value proposition:

> However there is a rampant fiction that if you supply a new safer method
> everyone will use it. For gods sake, the simplest of concepts like the stack
> protector took nearly 10 years for adoption, let people should switch
> languages? DELUSION.

This is mostly true. Developers in particular don't generally care about
security, so selling Rust as a "secure" language is not going to be enough.
I've said this since 1.0. But it's not entirely true for _products_ , which
often drive development - Chrome's pairing of performance and security led to
tons of marketing wins.

Given that tools like "cat" etc are:

a) Not generally security sensitive

b) Target developers

I don't see anyone choosing the "rust cat" over the builtin. This is why
people build tools like "bat", that _aren 't_ just memory safe copies, but
they're memory safe tools that add features and target slightly different
cases.

Not much else to get from this post, I think.

~~~
throwaway894345
Yeah. Not sure about BSD, but I was wading into building the GNU coreutils and
other GNU packages just yesterday. Fresh hell, they all seem to be build
dependencies of each other. ‘sed’ is its own build dependency.

The whole C ecosystem is a joke with respect to builds—all dependencies are
implicit; you’re just expected to have the exact dependencies installed on
your system at the exact versions and in the exact locations on the filesystem
that the build tooling is willing to look. Autotools and CMake are absolutely
terrible; they make Python’s build ecosystem look sane and well-designed.

So no, “security” isn’t the most compelling use case (for me, anyway), it’s
moving past these dumpster fire build systems as quickly as possible so mere
mortals can build their own software. Specifically, hasten the day where my
entire application dependency tree doesn’t bottom out in some Autotools,
CMake, shell script, project.

~~~
Uehreka
Agreed on a hundred thousand.

I used to feel sheepish defending npm and the whole node package ecosystem
against its critics, but once I started trying to run deep learning
applications or WebRTC media servers I quickly realized that some of the
critics are probably coming from a much worse package management system that
they’ve merely grown familiar with.

Want to have two pieces of software on one computer that rely on... idk, two
different versions of libwebsockets? Yeah, you’ll need either
docker/overlayfs, or some filesystem or pkgconfig hack that I’m sure a comment
reply will helpfully mention if you want two have to versions of a library
installed for two different executables.

~~~
Arch-TK
"two different versions of libwebsockets"

If your programs depend on different MAJOR versions of libwebsockets and
libwebsockets' build system doesn't easily allow that to happen then complain
to the developer of libwebsockets about it. If the developer of libwebsockets
is not interested in solving this problem then complain to the developers of
the programs which depend on libwebsockets that they should find a better
library which doesn't suck. If they don't listen, rewrite the tools you depend
on to not suck.

If, on the other hand, the programs depend on different minor versions of
libwebsockets and libwebsockets broke API in such a way that just using the
highest common version doesn't work then complain to the developer of
libwebsockets that they should stop doing terrible things like breaking API
and then complain to the developers of those applications that they shouldn't
let this kind of breakage slide and should move to a different library.

If you think this is unrealistic then I think your push towards insane
let's-pull-all-the-dependencies-into-one-binary/package build systems is
unrealistic.

You want a sane build system? The change starts when you stop tolerating
idiots who cause things not to work.

We don't tolerate engineers who build crappy bridges which break and kill
people. You shouldn't be tolerating programmers who build crappy libraries
which break API or don't allow two major-versions to be easily simultaneously
installed and linked against.

~~~
Uehreka
My issue is that the default behavior is to install these libraries globally,
even when they could probably be installed locally for the thing you’re trying
to build. (I picked libwebsockets out of the air, I don’t necessarily have a
complaint with that particular library)

On my old MacBook Pro I have dozens of node projects that I’ve downloaded over
the years just to mess around. If I ever want to clean up, I can delete a
project’s folder, and know that everything it installed is now gone (because
it would be in its node_modules folder).

On my linux desktop if I want to mess around with a cool project that’s
written in C/C++ I immediately pull out Docker and start trying to build up an
image, since I don’t want to pollute my global library folders with libraries
that I might be downloading just to try out this one thing. I can then delete
the docker image if I want to clean up. This works, but it feels like a
workaround to the main problem of libraries being global.

~~~
spion
The benefit of global libraries is that they are easier to update or patch. If
there is a vulnerability in an npm package, you will need to update it in all
your node projects (and/or rebuild all your docker images analogously).

A hybrid solution where the package manager is aware of all installed
versions, can symlink to a globally installed copy and can update all symlinks
would probably work.

Unfortunately, there is very little interest in fixing the problem on both
sides (distro PMs and language PMs) since they both believe their own approach
is right and don't care about the other group's problems.

~~~
pas
yarn uses a global cache and symlink if I remember correctly.

and npm has an "audit" feature, which is default on, so there's quite a bit of
movement in JS land to address the disadvantages of having a myriad small
packages.

~~~
spion
Its not about the size of the packages, its about managing an entire system,
for whatever definition of the word "system"

In the "olden" days, you would patch the vulnerable dependency and be sure
that all services running on the machine have been updated to the new version.
(in theory - if everything works great and there is ABI compatibility etc
etc).

Since the library is installed in a well-known system location, there are only
a few paths in the filesystem that you would need to check to make sure it's
up to date. This makes machine maintenance easier.

Nowadays, to patch a vulnerable dependency you need to potentially go to all
individual projects, check whether they depend on it, update the version for
that project, make sure to rebuild any containers that... contain it. Rinse,
repeat.

There is also not a single place where the dependencies live, so you can't
just run the same command across multiple machines to do the check - you have
to check the directory structure of every individual service.

This can be very slow and painful.

~~~
pas
Agreed. Yet it seems the ecosystem simply accepts this trade-off. For
increased productivity, increased memory-management security, etc.

------
bsdubernerd
It also doesn't help that Rust keeps on pushing on the idea that "static
linking is the only way to go". This is another cargo-cult which I wish didn't
end up being engrained so deep in the toolchain because while it has some
merits, it also has significant drawbacks of a typical unix distribution.

Static linking might be good for folks distributing a single server binary
over a fleet of machines (pretty much like Go's primary use case), or a
company that only cares about a single massive binary containing the OS itself
(the browser), but it stops "being cool" very quickly on a typical *nix system
if you plan to have hundreds of rust tools lying around.

Size _matters_ once you have a few thousands binaries on your system (mine has
short of 6k), think about the memory requirements of caching vs cold-starting
each, and running those and how patching for bug or vulnerabilities will pan
out. There's a reason dynamic linkage was introduced and it's _still_ worth it
today.

And rebuilding isn't cheap in storage requirements either: a typical cargo
package will require a few hundred megabytes of storage, likely for _each_
rebuild.

Two days ago I rebuilt the seemingly innocuous weechat-discord: the build tree
takes about 850mb on disk, with a resulting binary of 30mb. By comparison, the
latest emacs binary from git rebuilds in a 1/10 of the time with all the
features enabled, the build tree weights 150mb (most of which are lisp
sources+elc) with a results in a binary of 9mb, not stripped.

~~~
ddevault
How much do your programs _really_ share? After libc, libm, and pthreads, the
most common thing they link to is probably pcre, and I'm sure you can guess
how many of your programs are using that.

A good linker will shave off the parts of a library you're not using, and the
parts which are left over are usually not very big. The problem isn't with
static linking, it's that some "developers" think that bundling an entire
Chromium build with their app is a good idea.

Rust has a problem with big binaries (so does Go), but that's Rust's problem,
not static linking's problem.

~~~
legulere
Dynamic linking is pretty common in GUI applications (xlib, GTK/QT). Server-
side there's also libssl, xml libraries, zlib, curl.

~~~
lstamour
Outside of OS distro packaging, most Qt apps actually copy dynamically linked
Qt binaries to the same app folder for re-distribution to end users within
some form of installer or disk image. The same probably applies to Gtk apps
too, as I think the last time I manually installed Gtk for Windows for a Gtk
app was a decade ago.

Dynamic linking only works if you can guarantee ABI stability and folks
haven’t had to deal with ABI changes since C++13 to the point where if the C++
folks can’t change the ABI by C++23 we will forget it was ever a problem
because we’ll make the cost of change too hard. And the current C++ ABI
currently makes some parts of C++ executables sub-optimal unless you hunt down
your own standard-library alternatives.

Additionally, with dynamic linking, both code authors and code users now need
to agree on versions to support under assumptions that newer features in newer
libraries aren’t worth adopting quickly. To that end, some OSes do update
dynamic libraries more quickly, but doing so theoretically requires a lot more
recompilation and potentially you’re downloading the same binaries more than
once. At that point, dynamic linking is worth less than a binary-optimized
compression algorithm, no? Especially for distributing changes to said
executables.

Which isn’t to say, for OS distros, that dynamic linking is bad, far from it,
it tends to be the only valid solution for programming against OS core
components, but that in our haste to update dynamic libraries independently of
code compiled for them, we tend to forget ABI compatibility and the costs to
maintain API compatibility across a wide variety of dependency versions for
packagers and developers (or alternatively, the lack of updates to new library
features for end users).

Windows APIs never changing is the reason Windows stagnates more than macOS,
where Apple is less afraid to say your older app simply won’t run any longer.
Linux suffers less from this, but as pointed out in the email, part of that is
because POSIX implementations are relatively stable over decades, whether or
not significant improvements are still possible for more modern UX or
security, for example.

The details of dynamic linking on OS platforms can be found in this recent
series of posts:
[https://news.ycombinator.com/item?id=23059072](https://news.ycombinator.com/item?id=23059072)
(in terms of stability guarantees besides libc dynamic linking)

~~~
pjmlp
It is also one of the reasons why macOS is not welcomed by enterprise IT for
large scale deployments.

You can have dynamic linking with ABI stability with stuff like COM and UWP.

It is also the only viable way to do plugins in scenarios where IPC is too
costly.

------
dathinab
Let's try to summarize what he tries to say but kinda doesn't do very well:

_Rust and OpenBSD are not a good fit._

OpenBSD keeps to certain aspects about how to do thinks which just don't work
well with rust. This doesn't mean they are better or worse. _They are just
different._

For example outside of rustc-dev hardly anyone compiles rust them self. As
such not many work on making rust compile itself on exotic systems (and yes in
2020 i386 is a exotic system, even through it was the standard in the past). I
mean improving cross compilation is for now much more important (as far as I
can tell).

But OpenBSD requires the ability to self compile and support hardware for a
very long time so rust ( __for now __) doesn 't fit.

Another example is that many rewrites of classical small tools provide little
value to anyone _if_ they are 100% compatible. As such they tend _to not be
compatible_ in favor of better interfaces/output etc. (E.g. ripgrep!!)

Lastly OpenBSD is stuck in a world where thinks move very slow. But this is
also why many people use it. One the other hand rust is from a world where
thinks move much faster, sure with small steps and backward compatibility but
a small step every 6 weeks is still a lot over a fiew years. So they just
don't fit well together.

I also believe that in the future they might fit well together at some point.
But not for now and probably not in the next 3 or so years.

------
jcranmer
> I wasn't implying. I was stating a fact. There has been no attempt to move
> the smallest parts of the ecosystem, to provide replacements for base POSIX
> utilities.

This is in fact incorrect--there is a project aiming to build all of the
coreutils in Rust
([https://github.com/uutils/coreutils](https://github.com/uutils/coreutils)).

More to the point: while I do concur in the conclusion that Rust shouldn't be
a part of the OpenBSD base system, the gatekeeping implied here (it's not a
serious language because it's not used to build an operating system) is really
toxic. Especially considering that the gate in question has already been
thoroughly breached by the language in question (some universities have
switched to using Rust in their OS courses, for example).

~~~
cycloptic
There is no gatekeeping. To actually achieve POSIX-compliance is not an easy
task and requires lots of testing. If you don't do it then your replacement
will break everyone's scripts. A distro maintainer will also want to retain
compatibility with their supported GNU/BSD extensions so that's more work to
add on.

I would still agree with his statement at least as far as BSD is concerned.
Outside of Redox I haven't seen any serious projects to implement POSIX
compatibility. The Rust coreutils project is a good start but from their
readme it looks like they are more aiming to achieve GNU compatibility on
Microsoft Windows while being MIT licensed — I don't think anyone is seriously
using it to build a BSD (or even a GNU) distro. If I'm wrong about that I'd
love to hear it though. Rust is a good choice to write these things in but
let's be realistic about the time frame required to rewrite everything.

Plus I just downloaded those Rust coreutils and tried to build it and now I'm
waiting for 400 (!!!) Rust dependencies to download and compile. Is this
really appropriate for a core system component to have this many dependencies?
Or is there something I'm missing here? I admit I am not familiar with best
practices in Rust. As the project stabilizes I assume they will want to start
eliminating the dependencies or vendoring them upstream? At what point do we
decide to put these into a separate library like libbsd or gnulib?

~~~
DougBTX
> Or is there something I'm missing here?

There may be some platform-specific dependencies, but at least when building
on Windows, I saw 67 dependencies downloaded.

One thing that may be skewing the count is that internally coreutils is
packaged with one crate per command. There are about 100 commands, so you'll
have seen at least that many separate crates building. They're listed here:
[https://github.com/uutils/coreutils/blob/6e8c901204934029c88...](https://github.com/uutils/coreutils/blob/6e8c901204934029c883553bf8c69b9c43577714/Cargo.toml#L217)

My impression is that ecosystems where tracking dependencies is hard tend to
discourage them, while ecosystems where tracking dependencies is easy
encourage them. The coreutils crate depends on separate md5, sha1, sha2, and
sha3 crates, if managing external dependencies was hard I imagine those would
be bundled together. But it is easy, so it makes sense to organise the crates
so that each just provides one specific piece of functionality. Think single
responsibility principle applied to packaging.

~~~
gmueckl
Urgh, that's just a verification nightmare waiting to happen. Who is going to
wade through all that mess, review every package, pin its version etc...? You
need reproducability before you can get safety and security.

~~~
AgentME
Cargo uses lockfiles by default to pin all dependencies of an application, so
reproducibility shouldn't be much of an issue.

~~~
hedora
No one reads lockfiles. In python, people have successfully backdoored
projects by sending a PR that upgrades a dependency and regenerates the
lockfiles.

Somewhere in the 1MB lockfile patch, they pin a dependency to a version with a
known remotely exploitable flaw (sometimes a zero day, sometimes not).

------
zdw
As with most things Theo says, they're aligned with being the right answer for
OpenBSD's niche, which is probably not your niche.

Any new tools that have high resource needs, don't support all the more
exotic/esoteric platforms that it supports, are all nonstarters if your goal
is a very small core system that compiles quickly on all supported platforms.

~~~
all_blue_chucks
Yes but if he had an iota of social intelligence he would say "it is an
interesting project but it is not nearly ready for inclusion in OpenBSD for
reasons X, Y, and Z."

Instead he personally attacks people.

------
Datenstrom
A rewrite of core utils[1] was started in 2013.

    
    
        commit d4e96b33e343733992fad55ac840c9649cd72ede
        Author: Jordi Boggiano <j.boggiano@seld.be>
        Date:   Fri Aug 2 09:22:57 2013 -0700
    
            Initial commit
    
    

Although I hope that Redox[2] takes off more so than rust makes it into
existing systems. I dream of one day ordering a System76 with Redox already
installed.

[1]:
[https://github.com/uutils/coreutils.git](https://github.com/uutils/coreutils.git)

[2]: [https://www.redox-os.org/](https://www.redox-os.org/)

~~~
ashtonkem
I was going to say, all this discussion of whether you can make OS utilities
in rust seems to be ignoring the entire OS that’s actually being written in
rust!

~~~
jacquesm
How long until it will have the same kind of reputation that OpenBSD has?

~~~
ashtonkem
That’s really hard to say. Creating an OS that actually sees usage is _hard_ ,
and involves a huge number of non-technical hurdles to adoption. It’s entirely
possible that Redox will technically work, but never gain any following
because it doesn’t solve any problem well enough to justify the switching
cost.

------
wyldfire
> However there is a rampant fiction that if you supply a new safer method
> everyone will use it. For gods sake, the simplest of concepts like the stack
> protector took nearly 10 years for adoption, let people should switch
> languages? DELUSION.

Yeah, I agree: if the sole value proposition were 'security' it would be a
really slow roll. cargo is the killer app IMO. Anyone who's walked a big
dependency tree - download, build, oh oops this requires libfoo - download,
build, oh oops that requires libbar...

But I think lots of coders land on rust because they want or need something
faster or more portable than Python or JS and rust is 'easier' than C.

~~~
zerpelin68
Why is rust easier than C? Is it because it makes what is simple in C hard?

~~~
Ar-Curunir
Because it makes what’s hard in C easy: dependency management, cross-
compilation, memory safety, etc.

------
AzzieElbab
I have been using bat and exa along with alacritty for a while. All of them
are pretty good
[https://github.com/alacritty/alacritty](https://github.com/alacritty/alacritty)
[https://github.com/sharkdp/bat](https://github.com/sharkdp/bat)
[https://github.com/ogham/exa](https://github.com/ogham/exa)

~~~
jdxcode
Throw in those, + fd[0] and rg[1]

0: [https://github.com/sharkdp/fd](https://github.com/sharkdp/fd)

1:
[https://github.com/BurntSushi/ripgrep](https://github.com/BurntSushi/ripgrep)

~~~
AzzieElbab
Oh cool. Thanks

------
dorfsmay
This is from 2017.

In 2020 I personally use ag 99% of the time, and when I do use grep is because
its options are burned in my brain and it saves me 30 seconds by not looking
up the equivalent for ag.

[https://github.com/ggreer/the_silver_searcher](https://github.com/ggreer/the_silver_searcher)

~~~
jki275
Of course, and I use ag when I’m searching source code as well.

But ag isn’t a drop in replacement for grep, and when you have tooling that’s
depends on that, replacing it simply isn’t a viable option.

------
asveikau
Marc.info seems not to be responding for me. Here is an archive.org link.
[https://web.archive.org/web/20190302233853/https://marc.info...](https://web.archive.org/web/20190302233853/https://marc.info/?l=openbsd-
misc&m=151233345723889&w=2)

------
nromiun
> Adding such a toolchain would take make build time from 40 minutes to hours.
> I don't see how that would happen.

This should have been at the top of the email because this is the main issue
(along with the i386 one). As other people have noted in this thread some
people have already re-written basic utilities in Rust (and other languages).

Unless someone manage to make the Rust compiler much much faster then it is
currently it won't get accepted into OpenBSD anytime soon.

~~~
jacquesm
Yes, but are those rewrites really 100% drop-in replacements? That's a key bit
here.

------
protomyth
_Such ecosystems come with incredible costs. For instance, rust cannot even
compile itself on i386 at present time because it exhausts the address space._

Is this true anymore or just a history note?

~~~
hu3
> For instance, rust cannot even compile itself on i386 at present time
> because it exhausts the address space.

Last I checked Rust needed more than 4GB of RAM to compile itself.

~~~
Conan_Kudo
Yes, this is still true. :(

------
enz
This is not "de Raadt on Rust". There isn’t any de Raadt’s opinion on Rust.

This is Theo de Raadt on integrating Rust utilities in OpenBSD.

------
Zenst
Interesting read and raises one big issue:

We need to make memory safe languages accessible and with that, the tools and
other frameworks for a new language isn't an easy path.

Which gets to an aspect in many walks of life - if you are doing something
from scratch - you can use the latest knowledge and enforce that, but
knowledge moves on and bringing up the rest of the system inline with a what
is needed to use that new language/knowledge is not an overnight task, it's a
slow process - and even if you provide something better, you can not guarantee
it will be embraced at a needed pace to keep the momentum going.

Much like anything in life, you have better trains than you did when they was
invented, but legacy infrastructure precludes an instant/overnight switch, due
to impact upon what is used currently.

So case of, requirements for something better like rust support, is a bit of a
catch-22 as you need it there to get the people to use it so tools get
written, but to get it there you kinda need the tools so that people use it to
write the tools. Hence the first step is always the biggest.

Which makes you wonder how intertwined aspects of the OS and programs are,
even with standards like POSIX, there is clearly a bigger larger standard
needed.

This along with the dependances and you can see why docker and other
application level virtualization has much going for it.

------
SergeAx
> For instance, rust cannot even compile itself on i386 at present time
> because it exhausts the address space.

Wow, that one was really cruel.

~~~
pot8n
Seriously who cares about i386 in 2020?

~~~
thesuperbigfrog
\- Older computers still in use throughout the world.

\- Embedded systems and "Internet of Things" (IoT) devices

Many operating systems and toolchains continue to support i386:
[https://itsfoss.com/32-bit-os-list/](https://itsfoss.com/32-bit-os-list/)

Just because you do not use it does not mean other people do not.

------
beefhash
(2017)

------
dochtman
Perhaps relevant: Ted Unangst recently wrote some Rust code:

[https://flak.tedunangst.com/post/reliverator](https://flak.tedunangst.com/post/reliverator)

~~~
Mordak
There are actually several of the OpenBSD devs that have been writing Rust
lately. Some are doing it for work, some from personal interest. Opinions vary
on how much people like it.

Rust and OpenBSD share a lot of technical values though so maybe this isn't
surprising. Rust values safety and correctness. OpenBSD values safety and
correctness. Rust values 'Just Works' with all the nice things cargo does and
the whole 'fearless refactoring / concurrency' thing. OpenBSD values 'Just
Works' with sane defaults and the 'batteries included' base system.

~~~
jacquesm
Nothing wrong with the OpenBSD people writing in Rust, that's a step away from
tossing out say 'grep' or 'ls' and replacing it with something that isn't 100%
compatible. That will cause a lot of stuff to break in hard to predict ways.
It's like kicking out a random foundation stone from a building and expecting
no consequences.

~~~
Mordak
Sorry if you interpreted my comment as advocating for tossing out grep or ls -
that certainly wasn't what I was getting at. As one of the OpenBSD people that
happens to quite like Rust, I'm aware of some of the other devs who have been
using it, and we've talked a bit about Rust in general on occasion. The things
that people appreciate about Rust tend to align with the things they like
about OpenBSD, because the two projects have similar values. That's all I
meant.

------
raggi
Working on an operating system that uses some of these modern / safer
alternatives, the concerns around bootstrapping, lack of spec, lack of formal
memory model, long compile times, and larger output binaries are valid. They
are real and present challenges for us, that we choose to work against. It is
currently extremely difficult to produce a binary of the size of bsd cat using
these technologies, and very large amounts of work are needed to close that
gap, if it even can be.

In a new system you may not need cat though, but if you're a unix system with
a heavy shell and textual basis, you do need lightweight versions of these
things - light to build, light to run, and well integrated into the
environment.

What's the path out for unix systems? I have no idea, but at least as of today
I have not seen a modern safe language that is ready to replace _unix_ use
cases. Maybe zig? It's hello world is at least under 10kb.

------
BearOso
I want to like the rust language. The syntax is nice and the features are
interesting.

My problem is the ecosystem. The packaging and build system clearly isn’t
designed to accommodate classic desktop applications. The FFI is really
painful, and even Firefox has tons of wrapper code to just integrate with the
libraries it needs to have a GUI.

In that respect I agree with Theo in that Rust developers are focusing on far
too narrow of a use case, mostly web facing applications or data processing.
Cargo being so similar to npm kind of confirms this. Rust isn’t intended for
me.

------
maallooc
Feels like I've just had a coke. Nice reply from Theo.

------
eximius
He might just be wrong:
[https://github.com/uutils/coreutils](https://github.com/uutils/coreutils)

The only reason he might still be right is if they don't aim to be POSIX-
compliant. The page doesn't explicitly call out compliance as a goal, but
given that it has some busybox tests, it's probably compliant or very close.

------
ape4
Rust is good at preventing memory leaks. It doesn't matter (as much) for a
command line utility that runs for one second then exits. Of course, there are
now utilities written in Rust.
[https://github.com/samuela/rustybox](https://github.com/samuela/rustybox)

~~~
simias
I think Theo's reply was a bit excessive in some regards but I can also
understand him being frustrated when you've been working on a very complex
project for decades and somebody comes around and say "uh, why don't you
rewrite it in <hype new language>?".

It's seldom a very interesting or constructive discussion. If you think Rust
has something to bring to some project then you should at least take the time
to write a decent proof of concept or something similar, not ask the current
maintainers to rewrite their codebase in Rust/Go/Haskell/Prolog because of
some purely theoretical benefits. You don't go to construction workers in the
street and tell them that they should consider using different types of
hammers.

~~~
empath75
I don’t think Linux or bsd will ever be rewritten in rust nor should they. I
do think eventually there will be an os written in rust that surpasses them in
popularity, but probably on the time frame of decades.

~~~
qznc
There are not many CVEs for coreutils anyways:
[https://www.cvedetails.com/product/5075/GNU-
Coreutils.html?v...](https://www.cvedetails.com/product/5075/GNU-
Coreutils.html?vendor_id=72)

To improve security, Rust should probably aim to replace libjpeg and other
stuff which deals with malicious data a lot.

~~~
saagarjha
coreutils are often run on untrusted data.

------
lmilcin
There's something to it, but I don't thing the future is as grim as that.

Rust is a stepping stone. It is a proof of concept that shows a lot of
problems can be solved at compile time without adding huge bloat at runtime.

Yes, Rust has problems. Some of them are just because the language is new and
you feel not yet fully built.

The compile times are more more insidious side effect of Rust's properties. I
think, realistically, it will take some time but the problems will be resolved
either in future Rust or in another language based on it. And it will be very
worth the wait.

High level languages are fun and important, but it is not possible that all
system code is going to be built in high level languages. As we run stuff that
is more and more demanding of CPU we need a good low level language that will
also tick some important checks of reliability.

------
zozbot234
Interesting points. Would a rustc + cranelift solution manage to boostrap
itself in a 32-bit address space?

------
sgt
Theo is being conservative here. It may just save OpenBSD some 10 years from
now when Rust is no longer in the wind due to Hooch. At that point, someone
will propose rewriting coreutils in Hooch, yet it may be much more sensible to
stick with C, or rather C29.

------
stcredzero
BeOS/Haiku got itself into the 250 MB range of a full-featured distro, in part
by emphasizing code reuse in C++. (Though my understanding is that they used
GNU grep et. al.) The current macroeconomic climate is wrong for this, but
what if someone funded an effort to base an implementation of an entire OS on
Rust, let's say, using Linux as a bootstrap? So there would be no requirements
for "base to build base," and the developers dogfooding the toolchain would be
strongly motivated to make builds faster, which would be rolled back into
Rust.

~~~
smnplk
look into [https://www.redox-os.org/](https://www.redox-os.org/)

------
anewvillager
Please put a (2017) in title.

------
DominikD
This was discussed before here:
[https://news.ycombinator.com/item?id=15843631](https://news.ycombinator.com/item?id=15843631)

------
jason0597
I think it would have been good to include (2017) in the title. 2-3 years is
quite a long time in the area of technology, and in my view, Rust has come a
long way since 2017

------
otterley
(2017)

------
Koshkin
Hm. Rust needs more than 4GB to compile itself?

~~~
qayxc
Address space ≠ required RAM.

~~~
Conan_Kudo
It does require more than 4GB of RAM during its own build, though.

~~~
qayxc
That's simply not true. After I couldn't find any such requirement in the
build instructions for Rust, I was curious and actually setup a VM with 4GB of
RAM (3.8GB usable), checked out the current rust source tree and had my
machine compile the whole shebang in the background.

And what can I say - after about 3 hours and max memory usage of about 31% (or
a little less than 1.8GB) the compile finished without errors.

So could you please point me to the source of your claim? Did I just uncover
some magic loophole that allowed me to achieve this feat while using less than
2GB of RAM? What did I do wrong/right?

~~~
Conan_Kudo
We regularly hit memory exhaustion when building Rust on Fedora and openSUSE
for 32-bit architectures. It's even worse for openSUSE since we wind up having
to build LLVM at times too. What you're probably missing is building the debug
symbols. A release build with no debug symbols will likely be fine.

See: [https://github.com/rust-lang/rust/issues/45854](https://github.com/rust-
lang/rust/issues/45854)

~~~
qayxc
No you don't hit memory exhaustion. You hit __address space __exhaustion.
There 's a difference and it's an important one, because 64-bit architectures
don't suffer from this.

Your builds don't need more memory, they need more address space. You can
convince me otherwise by enabling PAE and still running into the issue. Until
then I stand by my point.

~~~
Koshkin
You are not wrong, but note that "memory" is not just RAM, it is also the disk
space that is needed for swapping pages in and out. So, the required address
space you keep referring to is not some cost-less abstraction; rather, it does
in fact correspond to some physical bytes - no matter where they reside at a
given particular moment, and it looks like the thing does actually need more
than 4G of those.

------
microcolonel
Re: bootstrapping the toolchain from the base system; we have mrustc now, and
it can build rustc. rustc uses LLVM for codegen, and LLVM is in the base
system now.

So at least that bit is there.

What I think would be a good candidate for first base-system rust component,
in case you are considering taking on a task like this, is a milter for
OpenSMTPD.

------
smcl
Hmm, this was back in 2017. I'm not an OpenBSD user nor a Rust developer -
does anyone know if anything's changed in the last 3 years (the memory usage
compilation issues, more mature/stable Rust-based coreutils, Rust compilation
times etc)?

------
t0astbread
Time and time again I'm astonished how much flamewar there is on HN when it
comes to discussing people's favourite languages. (Referring to the comments,
not the article.)

------
7532yahoogmail
Gate keeping is a thing ... And I smell bad odors here and there from the bsd
folks, but on the whole ... It's true: nobody is gonna trade across for equals
on grep, ls, whatever. If rust wants to be some OS dev language it's going to
have to make a better os or competive one (eg nextstep or k42 and similar
efforts) ... It's going to have to go much bigger. And finally _serious_ safe
code would be accompanied by frama-c / tla+ proofs .. just pointing out rust
is not the high ground.

------
vvanders
> As a response to this, Theo asked rhetorically "Where's ls, where's cat,
> where's grep, and ...

For my money ripgrep knocks the socks off of grep. I can't recall if it's
perfectly posix compliant but it's blazing fast and has a much nicer output.

~~~
creshal
From ripgrep's FAQ:

> Do you care about POSIX compatibility? If so, then you can't use ripgrep
> because it never was, isn't and never will be POSIX compatible.

------
jeffrallen
"Grim bloaty future" wins the Internet today.

------
metreo
Haha, Rust go brrr

------
zerpelin68
My hero.

------
woah
Is this guy trying to step up to Linus's vacated throne of "angry email
opensource guy"?

------
_wldu
Rust++ will solve this.

------
ryanlol
So whats the point of rewriting coreutils in rust? If you’re passing attacker
controlled inputs into grep you’re probably doing something wrong anyway.

It doesn’t seem like there’s much to be gained here. Network facing software
is where the big wins are.

~~~
damnyou
Speed. It's normally trivial to parallelize your programs.

edit: rough crowd today. Check out Send, Sync, rayon and ripgrep.

------
appleflaxen
>> As a response to this, Theo asked rhetorically "Where's ls, where's cat,
where's grep, and where's sort?"

> I wasn't implying. I was stating a fact.

It doesn't affect his point, per se, but this is a confusing way to engage the
topic.

------
yalogin
> Rust cannot compile itself.

This seems worth fixing. I think this is his main point.

There is also the bigger question if and why the core utilities should be re-
written in Rust. Given that safety is the main value proposition of rust it
may be makes sense, but it’s a much bigger discussion.

~~~
tomku
The "on i386" part that you stripped out when quoting is actually the
important bit. Rust does not care about having a self-hosting 32-bit x86
toolchain, while the lack of one is a deal-breaker for OpenBSD.

------
adamnemecek
If you can write C as well as the OpenBSD people, be my guest, keep writing C.
It might take you like 20 years to get somewhere interesting.

~~~
anthk
Like OpenSSH? PF? Clang? All of the modern compilers and languages made in C?

Maybe you can create something more widespread in Rust and not being a single
part of a browser.

~~~
pjmlp
Clang is written in C++.

The large majority of modern compilers and languages are either bootstraped or
written in a mix of the language itself and C++.

~~~
adamnemecek
And it took them forever to get anywhere. Compare LLVM with cranelift.
Cranelift made so much progress very quickly.

~~~
pjmlp
Indeed, but lets be honest about the optimizing capabilities and supported
hardware architectures of Cranelift.

------
empath75
All the core utils have been ported to rust for the redox project, and there
are actually superior alternatives in some cases available like ripgrep.

~~~
jeffbee
rg is great but it isn't a drop-in replacement for POSIX grep, which means
even if you have rg you still need to ship another grep so all your existing
scripts and programs that expect to be able to invoke POSIX grep don't break.

~~~
empath75
Yeah and there are multiple drop in replacements for grep— he was implying
those alternatives didn’t exist because rust wasn’t capable of handling them
or people weren’t interesting in writing them, but I think that’s been proven
wrong many times over at this point.

[https://github.com/redox-
os/extrautils/blob/master/src/bin/g...](https://github.com/redox-
os/extrautils/blob/master/src/bin/grep.rs)

~~~
otterley
Was this true in 2017, when the statement was written?

~~~
masklinn
Both redox's grep and ripgrep (which trivially demonstrates that a grep _can_
be implemented in rust even if it's not a POSIX grep implementation) were
created / started in early 2016, so yes.

Not to mention Theo's statement is so much weaker than actually successful
implementations the existence of these two completely blows through the goal:

> There has been _no attempt_ to move the smallest parts of the ecosystem, to
> provide replacements for base POSIX utilities.

~~~
otterley
What matters is when they were feature complete, not when the project started.
I can't plausibly claim that a program I wrote some code for but isn't at the
state of being a viable substitute yet is a "replacement."

(And yes, Theo, like many hackers, is prone to exaggeration.)

------
krick
> I wasn't implying. I was stating a fact. There has been no attempt to move
> the smallest parts of the ecosystem, to provide replacements for base POSIX
> utilities.

Huh? Some days all I see on HN fron page is "X written in Rust" projects,
where X is some long existing tool (now with colorized output by default!)

------
renewiltord
Every time I see a Theo de Raadt thing on the front page it's some nonsensical
language rant that has the utility of a PR review comment on some random leaf
JS project.

The guy does a lot of work maintaining OpenBSD but it looks like people only
upvote his pointless flaming. This is practically TMZ for Nerds stuff.

I think I'll block Theo de Raadt for myself on HN.

------
hkt
"For instance, rust cannot even compile itself on i386 at present time because
it exhausts the address space."

I don't want to be harsh, but who the hell is compiling anything on i386 now?
Real question, is anyone even making fresh i386 hardware that might be used by
consumers, or deployed in datacentres?

~~~
darksaints
OpenBSD is. Just because you can't think of a use case doesn't mean they don't
exist.

~~~
hkt
I'm asking about the quality of the reason. Is there a real reason to not
cross compile in 2020?

~~~
darksaints
Well, they're still supporting i386 releases, which means that there are still
i386 users out in the wild that aren't willing to upgrade their hardware.

------
darksaints
Admittedly it's not much of an opinion on Rust as a language but rather the
size and performance of its compiler. And I can understand why Rust advocates
would see it as a criticism of the language...because the language and the
compiler are heavily intertwined. I can also see that there are extremely
valid use cases for small and simple over big and complex, and in those use
cases, Rust is absolutely a bad fit.

I do, however, think it is odd that OpenBSD would value compiler size over
security, given the philosophical values of the OpenBSD project. I would
expect that sort of opinion (and agree with it) in the NetBSD project, but I
would think that OpenBSD would be willing to put up with big compiler binaries
and slow compile times in exchange for even small improvements in memory
safety. Strongly and statically typed languages in the ML tradition (like
Rust, SML, OCaml, Haskell) all have big bloaty compilers, but that is for a
specific reason: the type system that prevents you from compiling invalidly-
typed programs is in essence a test suite. And for rust, a language with a
type system that covers memory safety bugs, that test suite is _massive_.

OpenBSD is pretty famous for the comprehensiveness of its test suite, perhaps
only superseded in depth by applications that lack its breadth (such as
SQLite). I've never run the full OpenBSD test suite in a development cycle,
but I've heard stories about running it that perhaps parallel the frustration
newbies have with getting rust programs to compile.

What would an OpenBSD test suite look like if it could conclusively prove the
absence of all memory safety bugs? Would Theo De Raadt be willing to put up
with a test suite that ran for two hours longer and required more memory than
could be allocated on i386 systems if it could do so? I think he would...
_because that is something in line with the OpenBSD philosophical values_. So
why is it that it needs to compile code so efficiently, if you're gonna turn
around and run tests for 3 days to make sure nothing breaks?

------
013a
> Yes, now I am implying something: you won't bother to rewrite the utilities.

> So we cannot replace any base utility, unless the toolchain to build it is
> in the base. Adding such a toolchain would take make build time from 40
> minutes to hours. I don't see how that would happen.

So, to summarize: "We're not going to include the capability to write core
utils in Rust, and also, no one is bothering to do it."

Its rather startling to me how tail-wagging-the-dog, or maybe chicken-and-egg,
this situation is. Did anyone stop to consider that the reason why some of
these utilities aren't being rewritten is _because_ the unix old guard doesn't
want to support it in the toolchain? If this guy is the wall that you'll
eventually have to crash your car into, I'm not going to get behind the wheel
in the first place. Why waste my time?

Look; the compile limitations are real. I'm not saying that it _should_ be
supported. But, attitudes like the one are not productive. What is it about
operating system devs and these aggressive, mean, anti-social, self-important
personalities? A response worded like this should not be tolerated by the
community. It would have been just as easy and clear to say "We don't have
toolchain support for Rust, and we're really not interested in supporting it
due to its compile time and architecture support issues. <One sentence
explaining compile time issues>. <One sentence explaining architecture support
issues>. We can revisit this discussion in the future if/when these improve."

~~~
quotemstr
I got the same vibe from the email. It reads just like bigcorp turf defense.

When a smart but curmudgeonly component owner just doesn't want to accept some
outsider's change, there's no end to the technical objections that component
owner can raise against the outsider's change --- but these objections are
just a smokescreen, and addressing them is futile, because the _real_ problem
is that the component owner has done a thing certain way for a long time and
some attempt by an outsider to shake things up triggers all the owner's
ancient territorial defense instincts. The actual merit of the outsider's
proposal doesn't matter: the problem is social, not technical.

In a situation like this, when the intrepid outsider does manage to exhaust
the maintainer's objections, the maintainer will switch to just spewing
unfalsifiable FUD, to inventing further ridiculous objections, or just simply
ignoring the outsider's change entirely. There's no technical fix for a
curmudgeonly maintainer who really doesn't want to entertain new ideas.

In the corporate world, to resolve this problem, you usually have to go up the
management chain and explain to leadership that certain people are making
important work impossible. (You should try every possible alternative means of
persuasion first, of course.) In the FOSS world, you just have to fork as a
last resort.

~~~
jacquesm
What a load of bullshit. Theo has had more staying power than the Rust
community collectively to date as a maintainer of a mission critical OS.
That's a lot of responsibility and I personally very much value his defense of
what matters a lot to me: backwards compatibility. Not breaking stuff is the
prime directive at that level. Suggesting that your rewritten, non compliant,
non-drop in replacements and associated toolchain should be part of the
default is hubris at a level that does not befit nor benefit the Rust
community.

If you want to reach that level then you need to commit for real, and solve
those issues that Theo rightfully marks. Even though this was written in 2017
it is just as applicable today: either you replacement is as good on all
fronts as the thing it replaces _and_ has some additional benefits replacement
of something battle tested makes no sense.

