
Rust and the limits of swarm design - adiabatty
http://esr.ibiblio.org/?p=7303
======
ayrx
> This isn’t a question that comes up so often with respect to (say) Python
> because Python has an effective form of curation – blessing things into the
> standard library, at which point their alternatives generally disappear. In
> effect, Python modules are quality-filtered on the taste of the BDFL and the
> devteam.

I _love_ Python but this statement is absolutely coming from someone who does
not write Python on a regular basis. Many stdlib modules are greatly
outclassed by third-party alternatives - urllib2 vs requests, datetime vs
arrow just to name a few - to the point where no one actually uses the stdlib
modules in production.

~~~
rmattes
Sure, but even then, I think his point is still applicable. It's urllib2 and
requests, not urllib2 and requests and 20 other libraries. The fact that an
inferior library is being kept alive by being in the stdlib is maybe
unfortunate, but there's still only a small number of choices overall. That
makes it much easier to do research and see within the first few search
results that the majority of the community recommends one over the other.

~~~
ayrx
His point is absolutely not applicable. Rust is a _very_ new language. The
community will eventually converge on specific crates for specific purposes in
time.

------
oelmekki
Actually, the kind of features ESR asks for are covered by github, IMO. It's
easy to review code quality from there, the number of stars and forks reflect
project popularity, the list of committers (how many they are and who they
are) can help having an idea about long term maintenance expectations, etc.
The cool thing is that it works for almost any language : github is "social
coding", thus it's also social validation.

But this article raises an other question : if you want 10 years long
stability, is it wise to rely on third party code? If you plan on maintaining
your own project that long, it's probably acceptable to take time to write
your own implementation and maintain it. Or you could even use a third party
dependency, then progressively replace it with your own API compatible
implementation.

~~~
wott
10 years is "that long", now?

~~~
oelmekki
How many libraries you use today that you already used ten years ago? Is there
more than a handful?

~~~
jstimpfle
I don't think it's about "libraries that you use". It's about the codebase of
a widely used software. Most of the software I use existed 15 or even 30 years
ago. It has been updated, of course, but not necessarily redesigned.

------
nxrabl
>"A lot of Rustaceans don’t seem to grasp why, when the question is “where do
I get feature X?” the answer “oh, there are 23 crates for that” is objectively
terrifying."

This is a real problem. Here's a blog idea: "Consumer Reports for libraries"

(As an aside, the phrase "objectively terrifying" bugs the hell out of me.
Terror is fundamentally subjective - it's a primal emotion. Please let's not
let "objectively" become the new "literally".)

~~~
Manishearth
FWIW, this is not as bad a problem as he makes it out to be -- the reason
folks don't seem to grasp it's a problem is because from inside the community
the answer is usually "there are 2 crates for that, with the following
tradeoffs". You have to ask, however; and it's hard to find out which crate
you want to use without asking someone.

This is still a problem, however, but it's one the community has been aware of
for a while. [https://github.com/rust-
lang/rfcs/pull/1824](https://github.com/rust-lang/rfcs/pull/1824) is step 1 in
fixing it.

~~~
sidlls
"Inside the community" isn't a very strong argument in my opinion, because
there is a strong bias. Folks "inside the community" are developing Rust
packages because they really, really enjoy it. This is much different from
having a mature language where folks who aren't as gung-ho about it develop
things (out of curiosity, because work made them, etc.).

I understand there's a recognition of the problem, and I think it's great that
it's on the radar.

~~~
Manishearth
My point about "inside the community" was solely to say that it's not as bad
as he makes it out to be. You usually get recommendations and help picking one
within minutes of posting on a Rust forum. (Or by googling it). Survivorship
bias is irrelevant here.

This is not an ideal situation to be in, but it is better than he makes it out
to be. And of course, we're already working on fixing it.

------
LeanderK
i think he is exaggerating a bit. Many languages have a model where most of
the functionality lies in packages and not in the standard library. And it's
working. You just have to get used to it and try to develop a taste in
choosing the right package. But it's not fundamental and not the limit of
swarm design, because you can use statistics to assess maturity.

~~~
srssays
I wouldn't say it _works_ , rather that people will put up with it. When I
write JavaScript/node.js code, I have no idea whether the dependencies I'm
using will be supported 6 months from now.

It's not so easy to just "choose the right package". Often _all_ the packages
for a given function are someone's hobby.

Consequently JavaScript code needs constant, expensive maintenance. Whereas
Python code can rely more on the standard library and big frameworks and tends
to keep working for years.

Programming language designers should see themselves as builders and curators
of great standard libraries, the language itself is almost incidental to that
goal.

~~~
wwwigham
What makes the libraries blessed by the language designers so much better than
a library blessed by any other trusted party? Rust is an open source language,
and only the core team works on it full time - even then, any one of them
could leave at any time and the community at large would need to pick up the
slack. This isn't any different than the situation around any of the amazing
community libararies in the Rust ecosystem. Heck, libs blessed by the language
designers aren't even always the best choice - just look at json manipulation
in .NET; you get multiple different stdlib ways to work with json, yet the
community still gathers around the much better JSON.NET package.

In terms of value proposition, I think the author is just hankering for some
.NET style support agreements in order to hedge his maintenance cost bets -
and I'm sure there are or will be plenty of consultancies willing to provide
this service, so long as the demand exists. That open source language users
are entitled enough to expect this service for free from language
maintainers... I'm not sure that's completely justifiable.

~~~
srssays
If a library is supported and maintained by the language designers, then its
_lifetime_ will be at least as long as the lifetime of the programming
language. Or if the library is replaced, then it will be done through some
mechanism of deprecation, rather than sudden abandonment.

I don't see the standard library and third-party packages as being in
opposition. Far from it. A rich standard library encourages a rich ecosystem
of third-party packages.

Maintenance costs affect the authors of open source packages even more than
they affect commercial users. They are working for free and in their spare
time. A rich standard library makes a language attractive to people who want
to write packages.

~~~
bluejekyll
Omg. You just made me think about lifetime markers for crates... your project
requires a minimum lifetime of stability in a crate or something. Kinda funny.

------
taesis
It seems like the core thrust of this article is that someone isn't around to
perpetually support the libraries the author would like to use; i.e., he/his
company is unwilling to fork out the money for someone to keep the 'blessed'
libraries fully up to date -- so now that's the implementer's problem! I can
understand wanting stability and a promise of timely updates for all one's
dependencies, but it doesn't seem like the core language team is the right
group to foist that responsibility upon.

~~~
rntz
In the end, it doesn't matter who it's "fair" to assign responsibility to; a
language you can't rely on is a language you can't rely on. All esr is
pointing out is that there's a problem. Personally I think the Rust folks are
pretty aware of this problem, and are managing just fine; Rust is definitely
still in the early stages of language community growth, and locking things
down too much would be a mistake.

~~~
taesis
I suppose I'm not arguing from the perspective of fairness, but moreso about
incentives + experience. If we applied the author's argument against other
fields and languages, e.g. machine learning and python, then the cpython
maintainers would be right in including TensorFlow in the python stdlib (or
whichever competitor dethrones in and sticks around for long enough).

That doesn't sit well with me, at least. I'd want the top ML developers
looking at TensorFlow, and the best language implementers doing the same for
cpython, not the latter trying desperately to improve code they have no
expertise in. From that perspective, what matters is community: is there a
knowledgable community supporting codebase 'x', where x could be the stdlib or
some third party library. Some other commenters in this thread proposed better
library curation/discovery tools in place of a formal commitment of long term
support by the core team, which seems ideal! Using such an approach, you rely
on the language for the core-language bits, and on third parties for the
third-party bits, which seems on the surface to be healthier than arbitrarily
choosing the only constant group (core language devs) as your supporters of
everything.

------
moomin
Reminder that before you take anything ESR says remotely seriously, you should
read
[http://www.catb.org/~esr/writings/dancing.html](http://www.catb.org/~esr/writings/dancing.html)
from beginning to end.

~~~
jessaustin
He's not a Christian, so we should ignore what he says about computers? That's
preposterous.

~~~
moomin
That _really_ isn't a TL;DR; of that article.

~~~
jessaustin
Haha nice try, but one won't be drawn into a theological debate about e.g. how
many neopagan angels can dance on the head of an atheist pin. The point is
that religious beliefs, no matter how banal or how fringe, are completely
orthogonal to technical judgment. They are separate spheres. You've cannily
left unclear the precise nature of the spiritual pablum you're peddling, but
one can still see through your clumsy attempt to invoke religious prejudice.

------
epage
> Which brings me directly to what bothers me about the crate system and the
> sociology behind it – I don’t see any pruning. Worse, I don’t see much
> understanding of the need for it. A lot of Rustaceans don’t seem to grasp
> why, when the question is “where do I get feature X?” the answer “oh, there
> are 23 crates for that” is objectively terrifying.

Actually, I think the community does recognize that this is a problem they are
running into as they scale up. Previously there was the discussion of the Rust
Platform [1] as a pseudo-stdlib which was received with a lot of mixed since
the Python approach isn't actually all rainbows and sunshine. More recently,
there has been effort to try to improve the discoverability of relevant crates
[2].

[1] [https://aturon.github.io/blog/2016/07/27/rust-
platform/](https://aturon.github.io/blog/2016/07/27/rust-platform/)

[2] [https://github.com/rust-lang/rfcs/pull/1824](https://github.com/rust-
lang/rfcs/pull/1824)

------
pornel
I don't think Rust has a problem of having too many crates yet, and hopefully
crates.io will improve discovery before it becomes a problem. They're making
first steps to improve it already, e.g.: [https://github.com/rust-
lang/rfcs/pull/1824](https://github.com/rust-lang/rfcs/pull/1824)

------
sidlls
I can't say I agree with the author's first article (which he mentions in this
one), but I think he has the kernel of a good argument here. That's not to say
I agree with it entirely.

At some point "design by everyone" turns into "design by committee" by another
name. This criticism of Rust has manifested itself in other ways, too. For
example the amount of "unsafe" used in various crates (the idea being that
"the community" will collectively minimize "bad" use of "unsafe"). In my view
there's a tendency for some of Rust's advocates to simply assert that it will
work itself out without presenting good reason to believe so.

~~~
steveklabnik
Do you have any numbers on the amounts of unsafe? I would be interested in a
decent methodology to figure this out. I've seen people assert that this is
true in the same vague way you're referring to for the other perspective.

~~~
sidlls
I haven't any numbers and I wasn't asserting anything about the quantity that
exists.

~~~
steveklabnik
Cool. If anyone else has ideas here, get in touch.

------
galangalalgol
My company has to approve all software on the development network. Getting
newish versions of boost is bad enough. But keeping something like tokio or
even simd and all its dependencies approved and in sync and then transferring
them to the air gapped dev network is just this side of impossible. Is my
company an outlier or is this common in bigcorp?

------
dgellow
Maybe we could have something like what Debian has to manage channels?
Differents stages (experimental, testing, stable) and a group of official
maintainers? I know that's not easy to create that kind of communities and
that you née a lot of ressources (benevolent people). But that works great for
Linux distributions.

------
coldtea
> _My use case – foundational network infrastructure with planning horizons on
> a decadal scale – needs stability guarantees that Rust is not yet equipped
> to give._

Can you be any more pompous?

~~~
rst
He may be talking about NTP -- that is core infrastructure code that had been
poorly maintained for many years, and he's been involved in the rescue effort.
Points off for not saying so, though; without context, your reaction is pretty
natural.

~~~
bluejekyll
The thing is he is attacking Rust not based on the safety the language
provides, but more on a feeling that the package, crates, network doesn't fit
his belief of how things should be curated. He states that he thinks this is
why BDFL is good model, but gives no strong evidence supporting that view.

Rust has curation in its crate system. It's all in really good crowd sourcing
numbers. Crates.io, number of downloads, with trends (i.e. Even if it was
popular, is it still?). Then GitHub has great ones: stargazers == community
interest level; watchers == # of people interested in tracking the project,
possibly b/c it's a critical dependency; forks == # of people who've reviewed
the code; contributors == overall success in the community; commit graphs =
how much work has and is going into the project; issues can help you
understand if there are any critical underlying problems with the maintenance.

When I'm evaluating a library I look at all of these numbers to understand how
comfortable I am in pulling a dependency on the library. IMO this is far
better than the curation of some system level tools, which just become
defaults, and don't always offer users the ability to evaluate different
options easily, cron stands out in my mind as one that there are better
options than the default.

Anyway, his loss... the Rust community is awesome, _in my humble opinion_.

~~~
sidlls
> The thing is he is attacking Rust not based on the safety the language
> provides, but more on a feeling that the package, crates, network doesn't
> fit his belief of how things should be curated. He states that he thinks
> this is why BDFL is good model, but gives no strong evidence supporting that
> view.

One might easily argue that folks who criticize C and C++ for its ecosystem
are also "attacking" it, by your reasoning. It detracts from the rest of your
comment in my opinion (which isn't humble at all).

And "the Rust community" is not a monolithic entity. For example, I find "the
Rust community" in aggregate here on HN, other than a relatively small few, to
be rather caustic, aggressively defensive against criticisms of Rust, and a
bit too handy with downvotes for disagreement of legitimate arguments against
some of Rust's features, syntax, grammar, etc.

~~~
bluejekyll
'Attack' is a bit of hyperbole, so it's fair criticism, but I do feel that the
article he wrote had much less to do with issues with the language, but a very
monocular and subjective view of how ecosystems should be maintained.

He does have some good ideas in there about how to possibly to make crates.io
help you more easily identify the best crate for your usecase, but I think the
solution to that is a fairly detailed algorithm weighing the some list of
values like I put above. Otherwise you end up never allowing new options to
blossom.

Trying to read how what I put before wasn't humble, and perhaps your right on
that point as well. The perseption of humbleness is definitely subjective, and
the added color I threw in there probably in fact did remove any humility from
the overall statement, which of course are my own personal views and I highly
encourage people to question and criticize them.

~~~
sidlls
I meant my opinion wasn't humble, not yours.

