
Cargo: predictable dependency management - aturon
http://blog.rust-lang.org/2016/05/05/cargo-pillars.html
======
eridius
Something that I'm surprised this page doesn't talk about, and which is very
important considering the recent hubbub over left-pad, is that any dependency
you get on Cargo can be relied upon to continue to exist forever (well, as
long as the crates.io site still exists, but if that goes away so does the
Cargo index). The reason is because you can't ever remove a published version
of your crate from crates.io. You can yank a version, which tells Cargo not to
allow any projects to form new dependencies on that version, but the version
isn't actually deleted and any projects that have existing dependencies on
that version will continue to be allowed to use it. This is documented at
[http://doc.crates.io/crates-io.html#cargo-yank](http://doc.crates.io/crates-
io.html#cargo-yank).

~~~
incepted
> Something that I'm surprised this page doesn't talk about, and which is very
> important considering the recent hubbub over left-pad, is that any
> dependency you get on Cargo can be relied upon to continue to exist forever

Maybe the reason why it's hardly talked about is because it's common sense and
pretty much all dependency managers support it?

Except node of course, because they have no idea what they're doing.

~~~
eridius
Except not all dependency managers support it. I'm not even sure if it's safe
to say that a majority of dependency managers support it.

~~~
gnoway
Package persistence is not really a dependency manager feature though, it's a
package repository feature. npm didn't fail; npmjs.com did.

I'm just waiting for this to happen to bower next. AFAIK they're just a
registry pointing to github. All it's going to take is someone doing a force
push without thinking and we're in this same situation again.

------
dikaiosune
I like many things about Rust, but Cargo alone is what initially got me
started with it. I was in the process of setting up a fresh C++ project with
vendored dependencies, and it was just an absolute nightmare. In contrast, it
took me about 5 minutes to get a Rust project set up with comparable
dependency complexity.

~~~
akhilcacharya
Speaking of which, is there any good package management system for modern
C/C++?

~~~
dikaiosune
I've had biicode recommended to me
([https://github.com/biicode/biicode](https://github.com/biicode/biicode)),
but I haven't tried it yet.

EDIT: Also worth noting that a common response I've heard to "I need a better
manager for my dependencies" is "you have a system package manager for a
reason."

~~~
McP
What do they say when you reply "Windows"?

~~~
dikaiosune
I think it goes without saying that those with that sort of response tend not
to use Windows, and also often seem to assume that a perfectly in-order
Linux/Unix install is the only way to build software.

~~~
FreeFull
And also that you're fine with creating a package for any dependency that
doesn't have one.

~~~
dikaiosune
Creating, maintaining, shepherding through distro processes, etc. All so you
can then eventually build your own code.

------
dastbe
How does Cargo handle indirect dependency visibility? This article talks about
visibility in terms of "do I need to manually include indirect dependencies?
No!" but not in terms of "can I accidentally write code against an indirect
dependency?" which isn't sufficiently answered.

If the answer is that indirect dependencies are still visible, I'd be
interested in knowing if rust-lang/cargo plan to change that, similar to
JDK9's module system for re-export.

~~~
SimonSapin
Short answer: `extern crate foo;` doesn’t work if `foo` is an indirect
dependency. So you cannot accidentally use something without declaring it in
`Cargo.toml`.

Longer answer: when running rustc, Cargo passes individual `--extern
bar=/path/to/bar.rlib` options for each direct dependency, not just a path to
a directory with everything. You can see the exact command with `cargo build
--verbose` or `cargo build -v`.

------
xkarga00
I have been working as a Go programmer for the past two years and I really
love the language but dependency management has been one of the biggest pains
for me. Godep really sucks (that's what we use in my current gig - we plan on
switching to Glide, still haven't played around with it) and I am not sure I
like the fact that I need to go out in the wild and choose between the rest of
the existing solutions. I would prefer to have something that works ootb. The
Rust team really got this right and Cargo is amazing. One of the main reason I
have been dabbling with Rust lately.

~~~
dalailambda
In my experience Glide is the closest to a modern package manager for Go.

------
gue5t
Cargo assumes you want to cache downloaded/built dependencies at the
granularity of a unix user account. It's very easy to break things if you try
to force current cargo cache dependencies at the per-project level and there's
been no interest in caching things at a per-machine or shared-between-machines
level. If duplicating compilation work is desirable to ensure some amount of
noninterference, it should be supported to have a package cache per project.
If we can treat Cargo as a pure function from inputs to outputs, we should be
sharing these build results across the Internet, at _least_ within the set of
machines used by one person.

Cargo also refuses to follow the XDG directory specification for its per-unix-
user configuration; it should put things in $XDG_CONFIG_HOME and
$XDG_CACHE_HOME, but instead dumps both configuration and downloaded/compiled
stuff in ~/.cargo.

~~~
steveklabnik
You could also change $CARGO_HOME per project, to get a per-project cache.

~~~
BenTheElder
I actually do this with one project, with a very small script that wraps cargo
and sets $CARGO_HOME to a project local path before actually calling cargo. I
keep a very short cargo.py and cargo.sh with my project.

I've experimented with using this to "vendor" dependencies in a local .cargo
and so far that seems to work. An early example with discussion of the python
version is here:
[https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_bet...](https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_better_way_to_vendor_dependencies/)

------
yazaddaruvala
My Cargo wish list!

1\. Like every enterprise build system I've ever gotten to use, I wish Cargo
would manage the compiler version as well.

2\. The compiler forces me to mark `unsafe` functions/blocks to use unsafe
code. Why does Cargo not force me to mark `unsafe` dependencies to use unsafe
code?[0]

[0] Only dependencies that explicitly use `unsafe` should need to be marked in
my Cargo.toml. eg. Iron doesn't use `unsafe`, but Mio does. Iron depends on
Hyper depends on Mio. If I have a dependence on Iron, I really want to be able
to mark `#[unsafe_deep_dep] mio`, or else my build fails.

~~~
steveklabnik
At the fundamental level, the machine is unsafe. This means that if unsafe
were transitive, all code would need to be marked unsafe. Use a String? Your
crate needs to be marked unsafe.

~~~
yazaddaruvala
We agree. I think there was a bit of misunderstanding. I don't think
transitive unsafe is appropriate.

Only crates which use `unsafe` explicitly should need to be whitelisted.

Fun Example: I need to depend on Iron. Iron boasts in its README that it
doesn't use `unsafe`. Cool.

However, Iron depends on Hyper. Hyper is complicated, it needs to use
`unsafe`, I can understand that. Hyper is also used by a lot of people, I
trust it, and whitelist Hyper. My build passes, I know only `std` and Hyper
use `unsafe`.

Why is this helpful?

Hasn't happened yet but @reem, being human, adds an `unsafe`[0] and lets
Iron's README go out of date. It happens. However, as a user, I don't like to
trust documentation, and I don't like to trust other developers[1]. When I
update to this version of Iron, I'd rather my builds now failed. Then I can
decide if this escalation of responsibility is appropriate.

This is especially true in an enterprise context where teams of new hires do
whatever it takes to push features, and "accidentally" push some really unsafe
code to a deep dependency[2].

[0] Lets pretend this `unsafe` was actually used in production code:
[https://github.com/iron/iron/commit/ba4d197030067d4347134c2e...](https://github.com/iron/iron/commit/ba4d197030067d4347134c2e0897ef2fb4545353)

[1] Hence my other preferences for type checking, core reviews, and tests.

[2] Not everything is manually catchable in code reviews. Sometimes its 3am.
Sometimes whoever it was, was really new and the CR is 14 pages long and needs
to be deployed Thursday! Sometimes a reviewer is just having an off-day. It
happens.

~~~
yazaddaruvala
Another really cool thing is:

Even slightly discouraging the use of unsafe via Cargo will push crate owners
to find already trusted "miro-crates" which abstract their usage of unsafe.
Imagine something akin to left-pad existed solely to abstract unsafe. Most
crates already use it, so most teams already whitelist it. Using finding and
using a micro-crate might mean less friction for users to add your dependency
into their application.

Benefits:

1\. Smaller the crate, smaller the surface area, better the audit-ability.

2\. The more people using the same crate, the less probable there is a bug.

3\. If a relatively small crate exists, solely for its abstraction of
`unsafe`, and it is relatively popular, then that could be a good indicator
for moving that logic into `std`.

------
conradev
I appreciate that predictability is a big priority with Cargo.

I feel like Rust and Cargo are in a great position to deterministically (per
compiler version, per set of build flags) build libraries. That would be an
amazing step forward in security if it could be enforced. Does anyone know if
this is planned?

[https://reproducible-builds.org](https://reproducible-builds.org)

~~~
steveklabnik
In general, we are very interested in reproducible builds, and have fixed bugs
where accidental non-determinism has crept in. However, it can be easy to
reintroduce with build.rs, which allows for executing arbitrary Rust code
before a build. Syntax extensions are another problem here. But the scope is
much reduced, it's true.

~~~
conradev
That's awesome to hear. Would it be possible to make an attribute like

#[deny(non_deterministic)]

which would error if `build.rs` or similar is present? Also, for usage of
things like the `file!` macro, which might mess up determinism based on the
build directory.

Theoretically, syntax extensions could be written to be deterministic as well
(for a given version, of course).

Is there a tracking issue on this currently?

~~~
steveklabnik
There isn't, it might be an interesting idea! Not allowing build.rs would
eliminate a lot of crates, and not all build.rs' are nondeterministic... So
that's tough.

~~~
conradev
Yeah. Perhaps developers could attribute functions in `build.rs` specifically
with "deterministic", so you could see what in the dependency graph isn't
deterministic.

Unfortunately, unlike with borrowck, reproducibility is inherently hard to
verify, and you'd have to do it manually with a VM or some different build
environment.

This is interesting: [https://github.com/rust-
lang/rust/pull/33296](https://github.com/rust-lang/rust/pull/33296)

------
SimonSapin
It was a happy day when Servo ditched a big hairy pile of makefiles + git
submodules and switched to Cargo :)

~~~
Gankro
Happier than Rust 1.0 and the end of The Rustup?

~~~
cm3
End of The Rustup? AFAICS, it's still downloading a very specific servo-only
rustc.

~~~
kibwen
Servo's efforts to upgrade their version of rustc used to be legendarily
traumatic, these days it's comparatively tame. :)

~~~
cm3
Right, and if the binary toolchains would be built with musl (not targeting
musl) statically, we could have a rustc+cargo that works on both glibc and
musl, avoiding two versions.

~~~
dbaupp
Could you explain what you're referring to? It seems like a total non
sequitur, since as far as I know, there has never been any pain (or even
rumour of pain) in a Rust upgrade caused by what libc the compiler uses, but
rather breakage in the unstable features that servo uses.

~~~
cm3
If the official rust+cargo was built statically with musl, then a single
download would work on glibc and musl distros. Right now, we don't have a
download that works on musl, and one has to bootstrap on a glibc system,
targeting musl, which makes it impossible to bootstrap rust+cargo on a musl
distro without glibc because bootstrapping requires a snapshot.

~~~
dbaupp
Ah, it is a total non sequitur, unrelated to what servo encounters when
changing Rust versions?

~~~
cm3
I don't understand what you're saying. I suggested that linux builds of
rust+cargo ought to be done as musl-static (not for targeting musl) so that a
single download works on both Ubuntu and Alpine. Now we have only glibc
builds. Yes, it's not related to servo, but it is relevant to you mentioning
glibc, though this is off-topic so let's stop arguing.

~~~
dbaupp
I think you should go back and re-read the conversation, particularly kibwen's
comment above and your reply: they seem to be totally unconnected. I was
trying to understand how you thought they (Servo upgrading Rust versions and a
musl-compiled rustc) were related, I was not arguing. There is in fact _no_
mention of glibc until your comments, nor is there any implication in my
comments that I don't understand what musl does nor do I disagree that it is
useful (in fact, I used Rust's easy ability to link binaries against musl just
a few days ago). This discussion is _purely_ prompted by me trying to
understand why you thought it made sense to talk about musl on a seemingly
unrelated comment... maybe there was some insightful connection I missed, but
it seems not.

~~~
cm3
You're right.

------
cm3
It's great that Raph's line-breaking code has been reused, but in contrast to
Mozilla projects any contribution to it will now require assigning copyright
to Google (like Ubuntu or Microsoft's CLA require as well). At least it's a
tiny piece, so may not hurt too much in terms of prevented contributions.

~~~
gpm
Or just forking it, it's released under a open source license.

This wouldn't be the first time Servo has forked a dependency, for example
they maintain a fork of glutin (library that handles opening windows) because
they want certain features that upstream doesn't.

[https://github.com/servo/glutin](https://github.com/servo/glutin)

~~~
cm3
That can work.

Curious what features servo/glutin has that are exclusive?

------
Perseids
Something which is missing entirely in this concept (or, hopefully, only in
the blog entry), are security updates. For example, let's assume I use an HTTP
library whose TLS dependency has an error where it doesn't properly match the
domain name in the certificate to the hostname of the server I'm connecting
to. My Cargo.lock file references a TLS library of version 1.7.1, but the
security patch is applied on 1.7.6 which has some breaking changes. Thus, the
security update cannot happen silently (which would violate the predictability
property of cargo anyway). Instead, we need a command to selectively update
only packages with security updates and respectively we need a way to mark
version bumps as security critical.

------
shadowmint

        Libraries can be compiled as static libraries or dynamic libraries.
        even static libraries might want to do some dynamic linking (for 
        example, against the system version of openssl).
    

haha... I like cargo a lot, but I feel this is glossing over the biggest issue
building rust code has: Building and linking to native libraries.

    
    
        Similarly, applications are often built for different architectures,
        operating systems, or even operating system version...
    
        Compiling winapi v0.2.6 
        Compiling libc v0.2.10
    

Oh? I'd love to see that compiling on a different operating system.

Linking against the _system version_ of openssl? Are you sure it's the _right_
version of openssl not to break your binding?

The C ecosystem is fundamentally problematic to get repeatable cross platform
builds with; and trying to get repeatable cross platform builds with rust when
it touches _any c / c++ library_ is quite a pain.

Piston is a good example of how this is done... but people still struggle to
compile it; the 'uses the compiler to explicitly build the dependency as a
static library and links it' (eg. git binding afaik) is a much better
solution, but it massively increases build times.

It's a rough edge point; and I still feel like there's been no real progress
towards solving the issues.

Everyone just does something completely different in their native build
(build.rs) step, and then your 'repeatable build' is largely, hit 'cargo
build' and 'hope nothing goes wrong and all dependencies are installed'.

There are other more important priorities for rust at the moment (like
recovering from panics...), but this is certainly an area I'd love to see
improvement.

~~~
dbaupp
_> Oh? I'd love to see that compiling on a different operating system._

You'll note from the file paths in the blog post ("file:///Users/ykatz/...")
that the builds are actually being run on a OSX machine: winapi is careful to
not break builds when it is irrelevant.

It is possible and even necessary to write code that depends on operating
system details (e.g. to implement the high-level cross-platform code), and
indeed that code may not compile on different platforms. The point is that
rustc and cargo together provide the tools to make it possible and _easy_ to
write code that builds and runs everywhere everywhere, they
don't/can't/shouldn't guarantee it.

------
cm3
I see there is `cargo install` which will fetch and build an executable from
crates.io, is there no `cargo update` for `install`?

~~~
steveklabnik
[https://github.com/rust-lang/cargo/pull/2405](https://github.com/rust-
lang/cargo/pull/2405) landed, but [https://github.com/rust-
lang/cargo/issues/2082](https://github.com/rust-lang/cargo/issues/2082) is
similar too and is still open.

~~~
cm3
Interesting, but I was thinking more of something like "update installed
binaries with latest version from index". This doesn't look like it.

~~~
steveklabnik
Ah yeah, it's per-thing, not every thing you've installed, currently. cargo
install --list does list all the things you've installed, so it wouldn't be
too big of a deal to add; the pieces are all there.

------
Siecje
How does Cargo manage the case when dependencies X and Y both require a
different version of dependency Z?

~~~
steveklabnik
It attempts to unify them if it can, but if they cannot be unified, it will
include both versions.

~~~
cm3
Include both how? Make renamed copies to have two and modify the code to use
the new names?

~~~
steveklabnik
It doesn't have to do any renaming or modifying of names. Rust has a module
system, so each dependency will use the version they need.

The only time it doesn't Just Work is if one of those crates re-exports a type
from the sub-crate in a public manner, and then you try to call something from
the other crate with a value of that type. You'll get a compile-time error
about mismatched types.

~~~
cm3
But it sounds like that's what's going on. Using foo-1.1 in bar.rs and foo-2.2
in rabbit.rs.

~~~
steveklabnik
Sure. That will Just Work in every case _except_ if bar and rabbit both re-
export a type from foo, and you try to use them together. If they don't re-
export anything, then it all works just fine. No renaming or transformations
needed.

------
cetra3
This article purports that Servo relies solely on Cargo for the build step.

If you look at the build step from github it looks like it's relying on mach
also:

[https://github.com/servo/servo#building](https://github.com/servo/servo#building)

Shouldn't it instead be using cargo purely?

~~~
SimonSapin
Mach is what you interact with when doing stuff in Servo, but all the build-*
sub-commands end up calling Cargo, which is what does the heavy lifting.

The typical `./mach build` command will download known-good versions of Rust
Nightly and Cargo (unless that’s already done), call `cargo build` with some
flags, then show a system notification when it’s done. That’s it.

------
indatawetrust
the subject will be on the outside but what are examples of applications
developed with rust?

~~~
steveklabnik
There's a lot of OSS stuff as well, (well, "a lot" for how old Rust is) but we
just recently added [https://www.rust-lang.org/friends.html](https://www.rust-
lang.org/friends.html) to the website to showcase production users.

------
dschiptsov
So, dependencies in, say, cabal or go or CL are unpredictable?

The trend to put meaningless but catchy adjectives, like "safe" and
"predictable", as if everything else is opposite, is misleading.

Designers of classic languages were not some arrogant punks (they were
brilliant, like David Moon and other guys of his generation), and the most of
findamental problems were well understood in times of Standard ML or Common
Lisp.

So, in comparison to C or C++ it might be "predictable" and even more "safe",
but in comparison with well-researched classic languages, these mere redundant
adjectives.

~~~
dbaupp
By all reports ("cabal hell"), Cabal _is_ unpredictable, and Go's dependency
management problems prompted countless external tools until the recent
vendoring work: with the default `go get`, there is no guarantee that a build
will be the same from day to day, as the best one can do is point at a given
branch of a repository (a moving target), not tags or commits.

(I have no idea on CL's dependency management situation, maybe it is even
better than Rust's, in which case it would be _amazing_.)

~~~
f2f
minor factual adjustment: "go get" doesn't update a package once it's been
downloaded. you need to force it.

"there is no guarantee that a build will be the same from day to day" is
simply not true.

~~~
dbaupp
Oh, indeed. I must've got mixed up: it doesn't guarantee that builds will the
same from machine to machine (unless you run go get at the exact same
instant).

~~~
dschiptsov
So, in Golang there is no concept of package version?

For them different location (to import from) for an incompatible version is
good-enough.

~~~
f2f
go get is just a mechanism, not the tool. there are other tools build on top
of go get that ensure proper versioning.

