
The Rust Platform - anp
https://aturon.github.io/blog/2016/07/27/rust-platform/
======
tibbe
I wouldn't recommend following the Haskell approach. It hasn't worked well for
us. (I took part in creating the Haskell Platform and the process used to add
packages to it. I also used to maintain a few of our core libraries, like our
containers packages and networking).

Small vs large standard library:

A small standard library with most functionality in independent, community-
maintained packages has given us API friction as types, traits (type classes),
etc are hard to coordinate across maintainers and separate package release
cycles. We ended up with lots of uncomfortable conversions at API boundaries.

Here's a number of examples of problems we currently have:

\- Conversions between our 5(!) string types are very common.

\- Standard library I/O modules cannot use new, de-facto standard string types
(i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.

\- Standard library cannot use containers, other than lists, for the same
reason.

\- No standard traits for containers, like maps and sets, as those are defined
outside the standard library. Result is that code is written against one
concrete implementation.

\- Newtype wrapping to avoid orphan instances. Having traits defined in
packages other than the standard library makes it harder to write non-orphan
instances.

\- It's too difficult to make larger changes as we cannot atomically update
all the packages at once. Thus such changes don't happen.

Empirically, languages that have large standard libraries (e.g. Java, Python,
Go) seem to do better than their competitors.

~~~
pcwalton
I don't think most of these are applicable to Rust.

> \- Conversions between our 5(!) string types are very common.

> \- Standard library I/O modules cannot use new, de-facto standard string
> types (i.e. `Text` and `ByteString`) defined outside it because of
> dependency cycle.

We have one string type defined in std, and nobody is defining new ones
(modulo special cases for legacy encodings which would not be worth polluting
the default string type with).

> \- Standard library cannot use containers, other than lists, for the same
> reason.

> \- No standard traits for containers, like maps and sets, as those are
> defined outside the standard library. Result is that code is written against
> one concrete implementation.

Hash maps and trees are in the standard library already. Everyone uses them.

> \- Newtype wrapping to avoid orphan instances. Having traits defined in
> packages other than the standard library makes it harder to write non-orphan
> instances.

This is true, but this hasn't been much of a problem in Rust thus far.

> \- It's too difficult to make larger changes as we cannot atomically update
> all the packages at once. Thus such changes don't happen.

That only matters if you're breaking public APIs, right? That seems orthogonal
to the small-versus-large-standard-library debate. Even if you have a large
standard library, if you promised it's stable you still can't break APIs.

~~~
rtpg
But if you have a large standard library and want to break the API, you can.

If you have 100 different libs that are basically "standard" (who doesn't have
`mtl` in their applications at this point), now you have to coordinate 100
different library updates roughly at the same time. If you forget even one of
them, then you've broken everything.

I think the argument for a large Prelude/standard lib is similar to Google's
"single repo" argument: Easy to catch usages and fix them all at once. Plus
you're making the language more useful out of the box. People coming from
python can understand this feeling of opening a python shell and being
productive super quickly form the get-go.

Arguments for small std lib exist, of course. But Giant standard libraries are
more useful than not.

EDIT: I think the failure of the Haskell Platform has a lot more to do with
how Haskell deals with dependencies, and the difficulties it entails, than
with the "batteries included" approach itself.

~~~
hyperhopper
But even then, languages that have great, thriving easy to use dependency
systems and package managers with small standard libraries still run into
problems

see: javascript

~~~
wilmoore
The issue with comments written this way is that there are no details to
support the claim.

Writing "see: JavaScript" doesn't really help without context. Without
context, one does no know if you meant "JavaScript in browser" or "JavaScript
via Node.JS" or "I simply don't like npm".

I'm not claiming there aren't any problems; however, "problems" are
situational and one person's "problem" is another person's meh.

I just think it's irresponsible to not provide detail when making such claims.

------
yoshuaw
If a language is to play the long game, they must be conservative on what they
add. Even a minimal runtime like Node is still wounded by the addition of a
few broken interfaces into the core platform (even emitters, streams, domains
to name a few). These cannot be stripped out of the platform because they're
"blessed" and now everyone will _forever_ have a bad time. I suggest we don't
do that for Rust.

For a language to remain relevant in the long term, a system must be capable
of evolving. Going out there and blessing potentially conceptually incorrect
packages such as Mio is therefore not a good idea. The notion of "platforms"
best resides in userland, where collections can compete and evolve.

By keeping platforms / package collections in userland we can get organic
collections of stuff that make sense to bundle together. Imagine a "server
platform", an "event loop platform", a "calculus platform", a "programming
language platform". It would be detrimental to creativity to have all these
collections live in the shadow of the "rust committee blessed platform".

But so yeah, I'm not opposed to the idea of platforms - I just don't think
blessing userland stuff from the top down is the right play in the long term.

Tl;Dr: package collections sound like a cool idea, but if you care about the
long term, imposing them from the top down is bad

~~~
rcthompson
> These cannot be stripped out of the platform because they're "blessed" and
> now everyone will forever have a bad time. I suggest we don't do that for
> Rust.

It sounds like the proposal in the OP avoids this problem by having versioned
platforms that are independent of the Rust version. So if something turns out
to be a bad idea, it can be stripped out of later platform versions and
replaced with something better without disrupting users of the older
platforms.

~~~
vvanders
Yeah, I really liked how DirectX did versioning in that respect. Lets you
improve the API while preserving backwards API and signature. If you can give
Microsoft something it's that they do backwards compatibility well.

I don't see why that couldn't be worked into this idea.

~~~
rkangel
> If you can give Microsoft something it's that they do backwards
> compatibility well.

Completely agree. Don't underestimate the amount of effort it costs them
though.

------
Animats
I'm a bit worried when a language develops a "platform" and an "ecosystem".
This usually means you need to bring in a large amount of vaguely relevant
stuff to do anything. It adds another layer of cruft, and more dependencies.

Write standalone tools, but don't create a "platform". Don't make the use of
the language dependent on your tools.

C++ does not have a "platform". Nor does it need one.

~~~
pcwalton
C++ has innumerable _de facto_ "platforms" and "ecosystems". You have to
choose one to get anything done, whether that be various Boost libraries,
POSIX, Win32, Cocoa, Qt, GTK(mm), even stuff like XPCOM…

Helpfully, many of these platforms reinvent basic things like strings [1] and
reference counted smart pointers [2] in incompatible ways.

Wouldn't it be better if there were just one platform?

[1]:
[http://doc.qt.io/qt-4.8/qstring.html](http://doc.qt.io/qt-4.8/qstring.html)

[2]:
[http://doc.qt.io/qt-4.8/qshareddatapointer.html](http://doc.qt.io/qt-4.8/qshareddatapointer.html)

~~~
btown
And notably, C++11 actually moves in this direction, standardizing things like
smart pointers [0][1]. It's a very smart move for Rust. Core or near-core
library wars in the early days of adoption of a language leads to duplication
of effort, and for those invested in seeing Rust gain a set of libraries to
rival other languages, this is a great thing.

[0]
[https://en.wikipedia.org/wiki/C%2B%2B11#C.2B.2B_standard_lib...](https://en.wikipedia.org/wiki/C%2B%2B11#C.2B.2B_standard_library_changes)

[1]
[https://en.wikipedia.org/wiki/Smart_pointer#unique_ptr](https://en.wikipedia.org/wiki/Smart_pointer#unique_ptr)

~~~
glandium
Actually, C++'s STL is in a weird situation, compared to the standard library
in other languages, because the STL is a spec, not an implementation. And
there are as many implementations of the STL as there are compilers. This
might arguably happen if there were multiple Rust compilers, though. Anyways,
the result on the C++ STL is that in many cases, the same types have different
performance characteristics on different platforms, or worse, different
behavior/bugs.

~~~
aldanor
Nit: C++ Standard Library != STL

------
coolsunglasses
Haskell Platform is the last thing you should take inspiration from. Many of
us have been doing our best to kill it off. Maybe the downsides involved
wouldn't affect Rust in the same ways.

My suggestion, look at how Stack (the tool) and Stackage (the platform) work:

[https://docs.haskellstack.org/en/stable/README/](https://docs.haskellstack.org/en/stable/README/)

[https://github.com/commercialhaskell/stack](https://github.com/commercialhaskell/stack)

[https://www.stackage.org/](https://www.stackage.org/)

~~~
Ericson2314
The one thing stack does do nicely that this proposal doesn't is allow curated
comparable versions without forcing course dependencies. In plainer English,
stack can pick the versions while you still opt in pack by package. I beleive
that this is crucial to not slow down the evolution of the ecosystem.

In Cargo jargon, a solutuon would be for metapackages could double as sources:
`foo = { metapackage = bar }` to use foo from bar.

~~~
quodlibetor
To make this more concrete, you're suggesting something like

    
    
       [dependencies]
       hyper = { metapackage = { rust-platform = "2.7" } }
    

I like the general theme of having the platform be just a set of known-
compatible versions, but on the other hand this feels like it loses out on
many of the ease-of-use advantages of just specifying a platform version and
knowing that you have all the crates inside of it.

~~~
Ericson2314
Yeah that's a proper example.

I don't think we need to pick one way exclusivity. My specific use-case is
making PRs for packages to work in kernelspace / in unikernels. The library
night initially be packaged the easy way but then I'd use this. I don't want
the PR recipient to also worry they might get out of sync with the platform as
a side effect.

------
hsivonen
Having used Java and having experienced how you learn to replace the JDK URL
parser with something else, the JDK HTTP client with something else, the JDK
encoding libraries with something else, etc., I'm worried about entrenching
first-published libraries beyond their natural first-mover advantage and
making it even harder for better later-comers to be adopted.

OTOH, compared to e.g. C++, where even strings aren't standard across multiple
real large code bases (e.g. Gecko and Qt) having standard types especially for
foundational stuff can be very useful if the foundation is right or right
enough. Java having a single (though not optimal) foundational notion for
Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and
spec-wise correct (though not design-wise optimal) XML API (SAX) was a key
reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion
of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was
compiled!).

Still, at this early stage, I'd point to crates.io in general as the
"batteries" instead of creating a layer of official approval on top. But I'm
biased, because lately, I've been writing a second-to-market crate for a topic
for which a crate already exists.

------
qwertyuiop924
Good work! The idea of dropping extern crate is worrying, however. Most of the
ways that that would be done would add more irregularity, complexity, and
implied environment to the language (Rust Platform is always there, whether or
not you want it), all of which are opportunities for bugs to creep into code.

~~~
steveklabnik
What worries you about dropping extern crate? A lot of people feel like it's
duplicating what's in your Cargo.toml anyway.

~~~
qwertyuiop924
Cargo.toml isn't in language. It's for the package manager and the build tool.
The language shouldn't be married to either thing.

As for dropping extern crate, it either marries rust to cargo, or it makes
imports implicit. Neither is desirable.

~~~
tatterdemalion
One possibility is for rustc to take extern crate declarations as commandline
args and for cargo to invoke them based on what's in the toml.

~~~
Manishearth
This technically already happens

~~~
tatterdemalion
Right, I guess there's no info needed outside what the --extern flag contains.

------
hubert123
I mean really, why is this necessary? The reasons for adding this much bloat
are tiny and irrelevant. You get a big download full of packages you dont all
need, you dont know if you need and all for what? We all have Google, if I
want an HTTP library for Rust I'll google whatever the best one is and make a
judgement call myself.

~~~
sametmax
You can still have a tiny install if you want to. But most people not knowing
what to do will be able to have batteries included setup.

Basically: people knowing will still have freedom, people not knowing will
have an easier life.

And in the end, in 2016, you generally don't care if you download a few extra
100Mo. If you do, you will just find spend the time to find to have a smaller
setup.

~~~
majewsky
> And in the end, in 2016, you generally don't care if you download a few
> extra 100Mo.

I suggest a trip to anywhere outside the western world.

~~~
sametmax
I worked in Africa for a long time, I do know the value of a small dl. But in
that case, it doesn't prevent you from :

\- choosing a bare bone small manual install; \- making the download in
advance in a place where you can or during the night;

It's not like you have less option. You just have a clear default choice.

------
AsyncAwait
If the aim of the Rust Platform is to provide a 'blessed set' of 3rd-party
libraries, why not have something akin to an official, curated 'Awesome Rust',
where depending on what you want to achieve you can get a streamlined list of
well-maintained libraries, perhaps with user-submitted usage-examples,
comments, alternatives and the like - that way, you're not constraining Rust
itself to be in sync with all the 3rd-party libs, (and thus by necessity
stagnating to a certain degree), nor you're making authors of libraries that
are not in the Platform feel essentially invisible.

~~~
steveklabnik
How is that different than the proposal? The only part that's maybe different
is "feel essentially invisible", which is something that might happen, I'll
grant you that. But I don't think that people feel invisible when some package
is in a standard library, which is roughly equivalent here.

~~~
AsyncAwait
I just don't think that going beyond a simple website with 'blessed' crates
has much benefit, but it certainly has a lot of disadvantages as outlined in
this whole thread. I'm simply saying that if the goal is to have a central
place to point to when somebody asks, 'how do I do X', then perhaps making a
website/improving crates.io in this regard is the better approach for the
continued growth of the still very much evolving Rust ecosystem, rather than
creating a Rust "distribution" as such.

P.S. Thanks for all your work Steve, it's really appreciated.

~~~
steveklabnik
Thank you both for the compliment, and for elaborating. :)

------
Ericson2314
Combine this with my stdlib deps RFC[1], and I think we could _shrink_ the
nunber standard library crates versioned with the language! The standard
library metapackage would pull crates from crates.io as needed.

[1]: [https://github.com/rust-lang/rfcs/pull/1133](https://github.com/rust-
lang/rfcs/pull/1133)

------
cocktailpeanuts
Can someone summarize what's going on here for lay people? I know this "Rust"
thing is pretty popular on HN nowadays but not sure what the difference is
between a language and a platform. I was trying to read the article, but I
don't have enough knowledge about rust itself to understand what it's talking
about...

~~~
hueving
Rust is the language itself. The platform in this context is the standard
libraries you use for things like string manipulation, network connections,
etc.

~~~
dikaiosune
Pedantically, the standard library (strings, basic networking,
containers/collections, etc.) is already well defined and quite small. The
Rust Platform idea seems more aimed at providing access to curated, stabilized
versions of community-developed librares for common higher-level tasks (async
I/O, serialization/deserialization, etc.).

Personally, I think that letting users (especially beginners) opt in to a
larger set of starting libraries could be very beneficial for adoption.

~~~
themihai
Wouldn't be enough to have list of curated libraries documented somewhere in
the official rust documentation? It seems to me that a "platform" support
would encourage monolitic designs(i.e packages that work only within a
specific "platform")

------
jonstewart
The relationship between Boost and the C++ standard library might provide a
good example (especially in recent years, as there's been sustained focus on
expanding stdlib). Boost has a stringent peer review and comment period.
Sometimes I think they let in designs that are too clever, but they rarely let
in the clunkers that are seen in Python's standard library.

People then gain experience with it and sometimes subtle issues emerge. Those
issues can be fixed, either in Boost itself or when libraries move from Boost
to stdlib (and sometimes the issues inform language features).

While C++'s standard library is not "batteries included" like Python's, it's
been very gratifying to see it expand slowly and surely over the last few
years.

------
pc2g4d
Couldn't much of the function of such a "platform" be automated?

One of the main constraints that would guide selection of crates for this
platform metapackage is that they must have compatible dependencies. So
package A depending on Bv1 and package C depending on Bv2 wouldn't work
because they Bv1 and Bv2 would both have to be included, leading to a
conflict.

But this information is (in theory) encoded in semantic versioning. Assuming
proper semantic versions for crates, a target set of crates to be included
could be specified, and then automatically the various sets of crate versions
that do not have conflicting dependencies could be calculated.

These compatible crate/version sets could be automatically generated and
published as metacrates.

Consider the following crate/version dependencies:

    
    
      Av1 -> Bv1
      Cv1 -> Bv1
      Av2 -> Bv2
      Cv2 -> Bv1
      Av3 -> Bv2
      Cv3 -> Bv2
    

compatible_sets({'A', 'C'}) returns
{{'Av1','Cv1'},{'Av1','Cv2'},{'Av2','Cv3'},{'Av3','Cv3'}}

By imposing an ordering on these compatible sets they could be automatically
identified. compatible_A_C_0 is {'Av1','Cv1'}, compatible_A_C_1 is
{'Av1','Cv2'}, and so on.

Obviously the semver could be wrong and unexpected incompatibilities could
crop up. But couldn't these just be autogenerated and then voted on? Then the
top best compatible sets will filter to the top and, de facto, the Rust
Platform has been autogenerated?

~~~
steveklabnik

      > So package A depending on Bv1 and package C depending on Bv2 wouldn't
      > work because they Bv1 and Bv2 would both have to be included,
      > leading to a conflict.
    

As mentioned in this thread, this already works just fine. Rust can handle
both versions.

Furthermore, it's more than just a constraint problem; there's also
integration issues, testing the whole thing together, etc.

------
saysjonathan
I love seeing Aaron's work within the Rust community. I had the pleasure of
studying under his father and his family's gifts are clear in both their work.

Are metapackages going to be available for others to utilize? If so, how will
conflicts be resolved if packages require two different versions of the same
package?

~~~
brson
Yes, metapackages are intended to be a general cargo feature, available to
all. I suspect the design has not progressed far enough to definitively answer
your question about conflict resolution, but I'd imagine you have to override
that dep explicitly to fix it.

------
pc2g4d
The TeXLive of the Rust world? Could be helpful. But the small std probably
implies a constantly revolving cast of "best practice" libraries that's hard
to keep up with. I know in TeXLive there are no stability guarantees regarding
the collection as a whole. It's more of a collection than a platform.

------
xfactor973
I agree with the others here. I like rust the way it is and I think it's fine
to leave it alone :)

------
EugeneOZ
Good chance to see how Rust team will accept negative reaction to their idea.
If they will.

~~~
steveklabnik
Historically, there have been a number of proposals by the Rust team that were
received badly. We've always benefited from outside eyeballs on things;
usually the final proposals end up much stronger. We've only sought to
_increase_ this kind of thing over time.

------
moogly
For contrast, with .NET Core, Microsoft decided to cut up .NET's quite
extensive stdlib (BCL + FCL) into packages. On paper, it looks pretty good,
but it remains to see how well the versioning will work in the long run.

------
lindstorm
Any roadmap for Rust 2.0?

~~~
steveklabnik
There are no current plans for a Rust 2.0.

