
Comparison of Programming Language Package Managers - mustpax
https://docs.google.com/document/d/19HNnqMsETTdwwQd0I0zq2rg1IrJtaoFEA1B1OpJGNUg/edit
======
knucklesandwich
One of the things that I've found to be important for (and generally under-
documented by) package managers is resolution strategy.

pip, for instance, eagerly resolves the highest version of a dependency first
found in the tree and doesn't backtrack AFAIK [1]. Cargo sounds as though it
behaves similarly, but will actually backtrack [2]. I've also seen package
managers like 0install that will use SAT solvers to solve dependency
constraints [3].

I've found lack of backtracking on conflicts to be frustrating, but I'm
curious how the other two (and other conflict resolving strategies) compare in
practice.

[1]:
[https://github.com/pypa/pip/issues/988](https://github.com/pypa/pip/issues/988)

[2]:
[http://doc.crates.io/cargo/core/resolver/index.html](http://doc.crates.io/cargo/core/resolver/index.html)

[3]: [http://0install.net/solver.html](http://0install.net/solver.html)

~~~
haldean
pip's lack of backtracking is doubly irritating, because it installs packages
as it goes. It would be great if it would fetch all of the required manifests
and make sure that the install was going to succeed before it starts spewing
stuff everywhere, but instead it will happily back itself into a corner where
A->B->C(v1) and A->D->C(v2) conflict and so it leaves the system with A, B and
C(v1) installed and prints a nice red stack trace.

~~~
nilliams
Yep it's maddening. We use pip-compile [1] at work and check-in the then
totally-resolved requirements.txt. That is working for us well. However I
still head-desk every time I just want to install a local package (with pip
install -e) to work on it and pip runs amok, naïvely
downgrading/upgrading/doing-whatever-it-pleases :/

[1] [https://github.com/nvie/pip-tools](https://github.com/nvie/pip-tools)

------
halostatue
At least three things wrong with respect to Bundler/RubyGems:

1\. RubyGems is the package manager. Bundler is a meta-layer on top of that
which does full dependency resolution to find the right version before
installing. Bundler builds on the capabilities provided by RubyGems (and will
be integrated into RubyGems in the future).

2\. Bundler does support vendoring. It is widely discouraged, but I take
advantage of it in my app packager for deploys at work (Cartage:
[https://github.com/KineticCafe/cartage](https://github.com/KineticCafe/cartage),
specifically cartage-bundler [https://github.com/KineticCafe/cartage-
bundler](https://github.com/KineticCafe/cartage-bundler)).

3\. RubyGems and Bundler can use alternative sources (my work applications use
both public and private sources simultaneously).

One thing I don’t see that would love to see tracked and solved is
authenticity verification. RubyGems has support for signed gems but it’s not
widely used and hasn’t really been validated as Correct.

They should probably also look at CocoaPods and Carthage (iOS and macOS build
dependencies) and whatever Swift provides.

~~~
jacques_chester
> _It is widely discouraged_

It is also widely encouraged. Some engineers:

1\. Don't trust dependencies to always be there.

2\. Write apps that need to work in disconnected environments -- ie with no
internet connectivity.

I work on the Cloud Foundry buildpacks team for Pivotal. Being able to stage
and launch apps in a disconnected environment is A Big Deal for a lot of
companies.

~~~
halostatue
Absolutely. That’s sort of a fundamental reason why I wrote Cartage. I like
some of what Capistrano does, but the general insistence on being able to
`bundle install` on the target server is questionable, and the earlier
practice for Rails apps of keeping the gems in Git resulted in other problems.

Heroku mostly does this right with its 'slug' system, and I made Cartage
specifically to be able to make deployable packages that use dependencies. You
have a build machine that’s connected to the network and can reach RubyGems
(and, in the future with some plug-in work, other packaging systems—I expect
to add Node, Elixir, and Lua soon) and that will vendor the resulting build
and give you a tarball that is _just_ installable with all of the dependencies
of your package.

~~~
jacques_chester
Cloud Foundry has staging and run steps, much as Heroku does, to the point
that a lot of Heroku buildpacks will run without modification ... _if_ you're
in a fully connected environment _at staging time_. It's not an accident:
Cloud Foundry was in part consciously inspired by Heroku in the early days, so
adopting the buildpacks model was natural.

However, as I noted above, this model breaks for _disconnected_ environments,
in which neither the staging container nor the runtime container have internet
connectivity.

Heroku's ruby buildpack code runs bundler, Cloud Foundry's buildpack is a soft
fork of Heroku's, so either you vendor your dependencies _before_ sending it
to Cloud Foundry for staging, or you get a failed staging step when the code
in the staging container can't dial out to a remote repo.

~~~
halostatue
Yeah. That makes sense. My run steps are in Ansible, so it made sense for me
to make something that‘s just a packager. The nice thing about Cartage is that
it can make use of already-vendored packages (old way) or you can create the
deployment package from a developer machine. I really should write it up one
of these days, but I’m so busy getting my team’s pipeline fully fed that I
haven’t had time to properly shout out about Cartage (because, frankly, it’s
kind of awesome if I do say so myself).

~~~
jacques_chester
Cartage looks neat (thought Googling it took a few tries).

To me it "looks" like a buildpack, insofar as you are taking something,
injecting its dependencies and producing an artifact that's ready to run by
itself. If you ever whack bin/detect, bin/compile and bin/release onto it,
it'll probably work well enough as a buildpack in connected environments.

~~~
halostatue
Thanks. We’re going to add Node, Lua, and Elixir plug-ins soon to enable
standalone running. The nice thing about making just a tarball at the end of
the process is that it should be relatively easy to make anything else (Docker
image, AMI, etc.) with that.

[https://github.com/crohr/pkgr](https://github.com/crohr/pkgr) is the only
thing that’s _nearly_ similar, and it does a few things differently than
Cartage
([https://github.com/KineticCafe/cartage](https://github.com/KineticCafe/cartage)
for anyone else following this subthread). Pkgr is a little more opinionated
on using the OS environment and making sure that OS-level dependencies are
fully declared…I leave that to Ansible, for the most part.

~~~
jacques_chester
If you had an option to package the runtime, you could use the binary
buildpack[0] to run Cartage tarballs.

[0] [https://github.com/cloudfoundry/binary-
buildpack](https://github.com/cloudfoundry/binary-buildpack)

------
ryandrake
Call me old fashioned, but I find it annoying that every new language nowadays
seems to want to pull in its own package manager, redundantly parallel to the
perfectly good one my operating system provides. I am already perfectly fine
with installing my software through apt-get. Why oh why do I need pip, and
gem, and npm, and CocoaPods, and cargo, and NuGet, and on and on and on...

Using a new language is no longer a matter of cracking open a book, installing
a compiler, and firing up vim. You've got to change your whole lifestyle now.

~~~
knucklesandwich
Global installs are the root of a lot of headaches when building software in
the same way that global mutable state is the root of a lot of headaches when
developing it. Nix may be the one system package manager that is the exception
to this rule (I don't have experience with it, so I can't vouch towards that,
but I hear great things). However apt-get is incredibly bad at getting you the
dependencies you need for a reproducible build.

I've seen this argument come up a few times now, and I have a hard time
reconciling this rosy view of system package managers at the center of a
development process with my own experiences. That process was never as easy as
"cracking open a book, installing a compiler, and firing up vim" and usually
entailed a fair amount of trial and error, environment variable fiddling, and
sadness from having to chase down a derelict PPA (or the equivalent) hosting
the actual dependency version I need. Then more sadness as the number of
projects I work on increases and my dependency versions bifurcate.

~~~
jacques_chester
> _Nix may be the one system package manager that is the exception to this
> rule_

Some peers of mine recently experimented with building a Nix buildpack and
concluded that it won't work. Nix has hardcoded paths for its core
directories, and won't work in a security-constrained container.

~~~
trishume
I have no idea where you got that idea. Nix has built in support for building
normal non-privileged Docker containers. The paths are hard-coded but just
like hard-coded /usr/whatever paths in other unix software, that's what the
chroot in the container is for.

[http://lethalman.blogspot.ca/2016/04/cheap-docker-images-
wit...](http://lethalman.blogspot.ca/2016/04/cheap-docker-images-with-
nix_15.html)

~~~
jacques_chester
The paths that are hard-coded are non-negotiably incompatible with how Cloud
Foundry locks down its containers.

Other language managers don't hard-code and will allow stuff to run from a
local directory.

~~~
zzzcpan
They (other package managers) have to rely on at least a hardcoded elf
interpreter or they don't provide binaries and probably don't even handle
external dependencies.

~~~
jacques_chester
> _on at least a hardcoded elf interpreter_

Some rely on gcc, but again, it can be run from a local directory. Various
dynamic libraries can be looked up by setting environment variables or passing
parameters.

I think Nix is a great idea, but I'm not aware of an argument for hardcoding
those root paths which truly holds water. It makes sense of NixOS, sure, but
that's because it's an environment with top-to-bottom control.

The rest of us have to be good and share our toys.

~~~
trishume
Okay you should have clarified that in your original comment, it's not that
Nix is incompatible with unprivileged containers (it includes built in
support!) but only with a specific container system that isn't even the most
popular one (Docker AFAIK).

As for why those paths have to be hard-coded: they don't it's a config
variable you can change. When you change it you give up using other people's
binaries so you have to build everything yourself, which is exactly like every
other distribution system that relies on hard-coded /usr/whatever paths for
binaries. The FHS standard is incompatible with Nix's guarantees and dropping
it is where it gets all of its power, so it has to use something different,
and it chose to make that path /nix/store/whatever

------
munificent
I know Dart isn't super widely used, but it's package manager, pub, is
probably a good reference for this. We put a ton of work into it, and Dart
users regularly single it out as one of the compelling parts of the Dart
platform.

[https://www.dartlang.org/tools/pub](https://www.dartlang.org/tools/pub)

It hews pretty closely to Bundler's model (versions, version ranges,
lockfiles, shared dependencies), though. So I don't know if it gives you much
insight beyond "Yes, another language team things Bundler is on the right
track."

Personally, I have a hypothesis that states that all programming languages
either have a package manager that works like Bundler or will end up getting
one.

~~~
jacques_chester
Based on my work in buildpacks, I agree with the sentiment but I'm not as
hopeful as you are.

------
sivoais
Perl's CPAN has tooling for diffing versions through the use of MetaCPAN (a
top-notch site which every language should try to emulate). For example, here
is a diff of the URI distribution:

[https://metacpan.org/diff/file?target=ETHER%2FURI-1.71%2F&so...](https://metacpan.org/diff/file?target=ETHER%2FURI-1.71%2F&source=ETHER%2FURI-1.69%2F)

This information is also available through an API for integration into command
line tools.

~~~
mfer
IIRC, and I could be wrong, CPAN was the first to go down the route that many
modern toolchains now provide. We've looked at it.

In fact, the original creator of Glide (Go package manager) wrote about Perl
and CPAN when talking about Go at [http://technosophos.com/2015/09/02/dont-
let-go-be-condemned-...](http://technosophos.com/2015/09/02/dont-let-go-be-
condemned-to-repeat-past-mistakes.html).

~~~
sivoais
Yeah, I think it was. The only thing older was probably CTAN, but it didn't
have the same structure.

Other things from the Perl ecosystem that should be copied are:

\- CPAN Testers which automatically tests every package on multiple systems
from Windows to Solaris. This helps identify portability, backcompat, and
regression issues.

\- CPAN mirrors which ensure that there isn't a single point of failure. This
might not be as important now with fast networks and high uptimes, but it also
ensures that everyone can replicate all of the code at anytime. I believe R's
CRAN does this.

~~~
berntb
CPAN Testers also test on different operating system versions. And hence also
test the Perl binaries, for a lot of different versions, by literally running
the combined test suites for all of the (public) Perl code.

I really, really don't get why this isn't copied to the rest of the open
source language environments. (Masochism? :-) )

------
jbbarth
I don't get why "pip" is checked in "Has separate manifest and lock files".
Actually it doesn't have that feature (which Bundler for Ruby has, for
instance). This very feature comes with a third-party package called "pip-
tools", or possible alternatives, but raw pip doesn't have this ability
directly afaik.

Fwiw pip isn't even able to enforce versions correctly (packages are installed
as the file is read, and can conflict with previously expressed constraints).
Or report installed versions correctly (it's possible that packages are half-
installed or installed but not reported as such by pip commands).

~~~
calvins
Pip has a constraints file now. Running

    
    
      $ pip freeze -r requirements.txt > constraints.txt
    

after you've installed all your packages gives you a constraints file that can
be used to reinstall exactly the same versions:

    
    
      $ pip install -r requirements.txt -c constraints.txt

~~~
jbbarth
Didn't know that option thanks. Better than nothing, but unfortunately your
environment is still subject to the remarks in my second paragraph. So pip-
tools is still required if you want more guarantees.

------
steveklabnik
One thing about Cargo's lack of 'vendoring' on the checkmark list; there is
[https://github.com/alexcrichton/cargo-
vendor](https://github.com/alexcrichton/cargo-vendor) , and soon there will be
[https://github.com/rust-lang/cargo/pull/2857](https://github.com/rust-
lang/cargo/pull/2857)

~~~
jacques_chester
On the CF buildpacks team in NYC, lack of vendoring prevented us from
completing a buildpack that would work in disconnected Cloud Foundry
installations.

We were sad.

~~~
steveklabnik
Hm, even without the above stuff, `cargo fetch` should enable you to do
offline builds just fine.

~~~
jacques_chester
Can you point to a man page? We weren't able to find a reference doc on the
cargo site.

~~~
steveklabnik
"cargo --list" will show all commands, which shows fetch. "cargo fetch --help"
then shows the help for fetch.

We certainly have some work to do on Cargo's docs. Please don't hesitate to
open a thread on users.rust-lang.org if you run into some kind of issue. In
addition, I actually live in Brooklyn; if Pivotal ever needs a hand with Rust
stuff, just let me know, I'd be happy to swing by your office and talk about
whatever.

~~~
jacques_chester
Right, we've hosted you for a few tech talks (I'm that Australian guy who
MCs).

Email me on ... uh ... wednesday, maybe? We could get you to visit to check
the buildpack work we did last week.

------
anonymousguy
Since I had starting looking for an alternative to NPM I have discovered a
couple of things:

* All current package managers are either language or OS specific. What if you have an application with code written in multiple languages?

* NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.

* Some package managers do better than others with regards to managing packages. I found NPM encourages dependency hell and very little management tools for dependent or installed packages.

* A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.

I am trying to work on a solution to these problems at
[https://github.com/prettydiff/biddle](https://github.com/prettydiff/biddle)

~~~
mordocai
> * All current package managers are either language or OS specific. What if
> you have an application with code written in multiple languages?

guix and nix both work cross-language and cross-distro. Still OS specific
though since only linux AFAIK. Also, containers partially solve this problem.

> * NPM didn't have any kind of integrity checks for its packages, and I
> assume most package managers don't either. If you download a corrupt
> package, for example, you won't have any idea and it will still install.

Any package manager that doesn't do integrity checks is a bad package manager.
The only one I know of currently that doesn't is npm, but I haven't looked
deeply into every available package manager.

> * A lot of package managers seem to intermix packaging, distribution, and a
> registry. The registries tend to have limited names to pick from (like real
> estate) and can result in legal problems. Also if registration to the
> service catalog is required you cannot self-host or self-manage the
> distribution of your application.

What package managers don't let you self host? I'm truthfully not aware of
any. Even NPM does according to a quick google.

> I am trying to work on a solution to these problems at
> [https://github.com/prettydiff/biddle](https://github.com/prettydiff/biddle)

From your readme: "biddle is inspired by the incredible awesomeness of NPM".
Since NPM is literally the worst package manager I have ever used, that line
makes me want to stop reading and never touch biddle. I'd word it differently.

Edit: Reading biddle further. Dependency management and central hosting are
some of the primary reasons to have a package manager. At least for me, that
kills any interest at all. I imagine there's a niche market though?

~~~
nilliams
I often see drive-by bashing of npm, but rarely any justification. What
exactly is wrong with it?

(Aside from the lack of package integrity check, which I'll grant, sucks).

~~~
gnoway
I think it's not bashing of npm specifically so much as it is the node
ecosystem it serves and depends on; at least in my mind it's difficult to
separate node from npm. That said, for what it's trying to do (read a list of
deps, resolve vs. registry, download and unpack) it seems to do a fine job of
it.

My major complaint about npm is the choice to allow version range operators on
dependency declarations. We know the node.js ecosystem places a high value on
composability, so using lots of tiny modules which themselves depend on lots
of tiny modules is the norm. This is a problem though because range operators
get used liberally everywhere, so getting reproducible builds is like winning
the lottery.

There are other things I don't like about using npm: node_modules/ is big and
has a lot of duplication (even with npmv3), it's pretty slow, historically it
has been unstable, its still crap on Windows, etc. - but for someone who has
'ensures reproducible builds' as part of their job description, the way its
modules get versioned is its worst feature.

~~~
nilliams
For reproducible builds (or at least 'to get the same versions again') you
should be using 'npm shrinkwrap'. (Of course there's probably more you should
do to get true reproducible builds, but that goes for any package manager).

The range operators are important, else you'd never be able to resolve 2
packages that want a similar versioned sup-dependency e.g. jquery 1.12 because
without range operators those 2 packages would have declared minor version
differences (1.12.1 and 1.12.3) depending on when they were published. This
would mean you'd always end up with duplicated dependencies.

I'd argue 'node_modules is big' is not a fault of npm. If the package or app
you're trying to install generates a large node_modules dir, that is something
you should take up with the package maintainer. See buble vs babel - buble has
a way smaller dep tree.

npm is only slow in the ways that all other package managers are, when
installing large dependency trees or native dependencies (like libSass) and it
is way faster than say pip and rubygems in this regard. When I 'pip install -r
requirements.txt' at work, I literally go and make a coffee.

Also never experienced any instability, though I may have been lucky.
Certainly it has been very stable for the last year or so when I've been
working with a lot. Could you elaborate on why it is crap on Windows? I did
think all major issues (e.g. deep nesting problem) were now fixed ...

~~~
mordocai
The main problems we ran into with shrinkwrap were:

It shrinkwraps everything in your current node_modules directory.

This includes platform specific dependencies that may not work on other
platforms but now will cause npm install to fail instead of just printing a
message about it.

So our current workflow has to be:

1\. Update package.json 2\. rm -rf node_modules/ 3\. npm install --production
# This doesn't include any of those pesky platform specific packages 4\. npm
shrinkwrap 5\. npm install # Get the dev dependencies

As far as the other comments about npm, I just generally have more problems
with it than rubygems/bundler and the general OS package managers.

~~~
nilliams
Ah okay, I've never used shrinkwrap across platforms (dev in Linux or Linux
VMs, deploy to Linux). That does seem like a PITA.

> As far as the other comments about npm, I just generally have more problems
> with it than rubygems/bundler and the general OS package managers.

I generally don't :)

------
ericlathrop
I'd like to see how one of the oldest package managers, Perl's CPAN, stacks
up.

~~~
riffraff
It's in the list, and does better than basically everything else.

------
nilkn
This should probably use Stack for Haskell as well, which was designed to
solve various common issues that arise when using cabal as a package manager.

~~~
moosingin3space
Is stack really a package manager? It seems to me that it's more of a build
manager, delegating package management to cabal/stackage.

~~~
nilkn
I see your point, but I'm not sure if it matters. Stack might be built on top
of cabal, but I never use cabal directly at all anymore, and stack cleanly
addresses a _lot_ of problems that you'll run into if you exclusively use
cabal-install for package management. Stack actively installs packages and
dependencies on your system.

------
throwaway2016a
I'm confused as to why composer (PHP) got an X for "Central package
repository"... isn't Packagist a central registry? And other package managers
allow packages from source.

I'm not arguing the point just trying to understand the decision.

~~~
jadell
You're not actually downloading the package from Packagist. It acts only as a
metadata repository that points you to a source repository. For the others, I
would assume they act as both a metadata and a source repository.

~~~
throwaway2016a
Thanks for the clarification. I'm shocked it is the only one like that.

~~~
mfer
It takes a bit of work to store versions and manage them as a central repo.
Composer will fetch the sources, as a tarball from GitHub if possible, and
cache them locally. They are skipping being the central store and deferring to
root sources.

They removed a lot of complexity by doing that and I bet more than a few
people didn't notice.

------
stewbrew
Which one allows to import multiple versions of a lib/module/package with
different versions? (Not primarily but also an issue of the package manager.)

~~~
olalonde
Npm allows that.

~~~
moosingin3space
How do different versions of the same package interact with npm? Since JS is a
dynamic language, it seems that there could be problems if a function in one
version of a package returned a value which was then fed to a function in a
different version.

~~~
stewbrew
You typically import it like:

    
    
       var foo = require('foo');
    

The lib's functions are accessed solely via the foo object. There is no
interaction when different libs require different versions of a lib because
the access the lib via different objects/modules. This actually is the one
aspect of javascript I wish other languages would copy.

------
w4tson
Surely if a package manger provides the ability to store in VCS then it's
admitting defeat for reproducible builds. I'm thinking npm here

------
the_mitsuhiko
Cargo supports vendoring and enforces semver. Not sure why that document
disagrees.

~~~
steveklabnik
The difference is in the central repository: Elm will compare your types to
the previous versions and how the versions diff, and refuse to let you upload
a package that makes a backwards incompatible change without bumping the
version.

At least, that's my understanding of what they meant.

~~~
halostatue
Elm also, according to a Changelog podcast I just heard, does not allow
releasing below version 1.0.

