
Linux package managers are slow - ericdanielski
https://michael.stapelberg.ch/posts/2019-08-17-linux-package-managers-are-slow/
======
Asmod4n
Compared to Windows, they are all blazing fast. I still wonder how it can take
over an hour just to search for the latest Updates on a Windows 7/8.1 and/or
Server 2012 (R2) Box.

~~~
KingMachiavelli
In addition to taking hours, the Windows update is terrible opaque. Between
the 'old' update & new update interfaces, the time spent 'looking for updates'
can range from a minute to over an hour which means it's obviously doing more
than just querying a server for available updates. During the 'installing'
updates phase, the percent complete will stall & jump at random intervals
without any indication of why it suddently went from 15% done to 25% done.

And at the end of this long and ardouous process, you only have an updated OS
and still have to manually (or use a tool like Chocolatey) update your non-MS
software.

It baffles me that this as considered acceptable considering these issues are
still present even in domain environments with local WSUS servers.

~~~
mehrdadn
I'm not entirely sure about this (someone please correct me if I'm wrong), but
it seems to me that on Windows the problem is fundamentally harder than on
Linux. On Windows, you can (or at least could on Windows 7, where this time
spent was particularly awful -- though I think you can still uninstall
individual updates) pick and choose which updates you install (and uninstall).
The sheer number of combinations of updates installed is exponentially more
than anybody can test, yet that flexibility is given to you. On Linux you
don't seem to have this kind of flexibility as far as I can tell. Now,
precisely _what_ computation has to occur that's so much more work in Windows
than on Linux is unclear to me, but it does seem like Windows has to solve a
fundamentally harder problem.

~~~
Arnavion
To clarify what mehrdadn is talking about, two Windows hotfix updates to the
same component can be installed indepedently, or one can be installed and the
other not.

This ability does not exist in Linux package managers since they deal with
packages - there is only one newer package to install, and installing that new
package replaces the entire existing package. Hotfixes work by applying
patches to the affected components' files, which is why they can be
independent of other hotfixes applied to the same components.

Hotfixes are also grouped into larger updates, the largest of which is a
service pack. (You released N hotfixes over the last month and expect everyone
to apply them in the future. There's no point making every user in the future
download N updates where one batched up one would suffice.) So when Windows
Update has to answer the question "Is this new update required or is it
already installed via a previous update?" it has to do a bit of gymnastics to
figure out the answer.

So both approaches have their pros and cons. The hotfix method means you can
get targeted fixes with no feature updates and smaller chance of regressions,
at the cost of being more complex to solve than "if server's version >
installed version, download and overwrite".

~~~
Arnavion
For example, consider that users of component Foo discover that it crashes
when it tries to access the internet with BarCorp FireWall 2019 installed.
There might be a hotfix released that fixes Foo to not crash.

Since it only affects Foo users that are also BFW users, it might happen that
the hotfix happens to break Foo users who have QuuxCorp Internet Optimizer
installed instead. The testing might not have caught it since the primary
purpose of the testing was to ensure that the issue is fixed for BFW users,
with minimal sanity testing that it doesn't break non-BFW users.

So QIO users can mitigate themselves by uninstalling the hotfix. They didn't
need it anyway. Eventually another hotfix may be released that supersedes the
first one and is tested for both BFW and QIO, and this latter hotfix will
eventually be bundled up into a general Foo update.

All this happens independently of _another_ Foo hotfix released after the BFW
one, to fix the issue where it crashes when you open a .txt that's actually a
.bmp.

The equivalent with Linux packages would be to have .1, .2, .3 package
releases, but QIO users can't can't keep the changes in .3 and remove the
changes in .2.

~~~
KingMachiavelli
Some distros allow give you some flexability. RHEL as AppStream which allows
you to pin particular sotfware to a specific version while upgrading the rest
of the system.

The equivilent of Windows SxS would be like running everything in it's own
docker container. Assuming each one was updated concurrently & they used a
shared package cache this could still be quite fast compared to Windows.

Even accounting for Window's 'impressive' ability to let you install any
combination of updates/hotfixes. It's basic software engineering that the
common case should be fast and considering that in 99.99% of Windows instances
are going to have all of the updates installed, it's very apparent that the
Windows update system & SxS were pooly designed and is an major area at
Microsoft suffering from brain drain. I feel the most likely case is that no
one currently at Microsoft (and still working as a programmer as many of the
old talent are either retired or upper executives) understands it enough to
fix the issue.

~~~
Arnavion
AppStream still replaces entire packages. It does not work like the hotfixes
that I described.

WinSxS is not about hotfixes either, but for having different versions of
components simultaneously installed.

Neither of these are relevant to the point of this subthread.

------
Arnavion
I don't mind packages being installed sequentially, but I do wish they would
be _downloaded_ in parallel. It's very annoying to watch ~1500 packages, most
of which are only a few KiB in size, download at the rate of one per second on
a gigabit connection. (I'm on openSUSE, so zypper. I think yum downloads in
parallel now after dnf improvements were merged into it?)

~~~
orev
I think this is done more for the sake of the server sending the files, not
the client side. Most Linux distros are using some form of donated mirror
servers, and it’s not nice to slam them with requests.

~~~
Arnavion
That probably was the reason when package managers were invented. I don't
think it makes sense today, especially when we _do_ have some package managers
downloading in parallel (like dnf as I mentioned). That means at least some
distros believe their infrastructure can handle it.

As for volunteer servers, if a particular server can't handle >N connections,
it can drop them. No reason to gimp clients instead.

And specifically in openSUSE's case, nobody has put forth server bandwidth /
limits as a reason not to parallelize downloads. [1][2][3] It's just not been
done because nobody (including myself) has cared enough to figure out how to
do it.

[1]:
[https://features.opensuse.org/307862](https://features.opensuse.org/307862)

[2]:
[https://github.com/openSUSE/zypper/issues/104](https://github.com/openSUSE/zypper/issues/104)

[3]:
[https://github.com/openSUSE/zypper/issues/160](https://github.com/openSUSE/zypper/issues/160)

------
matthewbauer
\- The "Pain point: no concurrency" thing seems wrong for Nix. By default, Nix
isn't using any hooks at all, and it should be concurrent. The confusion might
come from the fact that NixOS does have something kind of like hooks, but
nothing makes you use them.

\- The QEMU results are highly misleading, because there are tons of different
configurations possible with QEMU. Nix builds all QEMU architectures while the
Alpine one only is getting qemu-system-x86_64. It's apples to oranges.

\- The Nix docker image is not actually running NixOS, but instead running
Alpine. It just has the Nix package manager installed on top of that.

------
doubleunplussed
Pacman aint slow. Seems to basically run at the speed required to download and
copy files. Then again it has no concept of required versions, since Arch just
requires you to stay up to date. So there's no extensive solving of
dependencies over version requirements, just the packages themselves.

When I was on Ubuntu though, which does have version requirements taken into
account in packages, I didn't find apt to be slow either.

You know what's slow? The conda package manager. Or installing software -
practically any software - on Windows. I regularly budget half an hour to an
hour just to install something on the Windows computers I'm forced to use.

~~~
phiresky
The main performance problem in pacman as far as I can tell is really slow xz
decompression. This is mostly important for large packages:

    
    
        $ time xzcat /var/cache/pacman/pkg/cuda-10.1.243-1-x86_64.pkg.tar.xz >/dev/null
        1:17,33 total
    

Just decompressing that 1.8GB package takes 77 seconds.

There was a discussion in April to switch compression to zstd - which reduces
the decompression time for that package to only 4 seconds (!) while keeping
the same file size. That way the disk is the bottleneck again, as it should
be. That switch would be awesome, but it looks like it's not happening
currently.

~~~
sigotirandolas
If you have a fast connection, xz decompression may be the bottleneck, but in
some places, downloading that 1.8GB can easily take an hour or more. So I’m
not sure whether this is the bottleneck in the usual case.

I would also be interested to see if it’s using parallel decompresssion when
possible. I have an intuition it doesn’t, since I have always found Arch
defaulting to single thread decompression, e.g in makepkg or for compressing
with mkinitcpio.

------
lidHanteyk
The author is probably clueless, not dishonest, in this particular mistake,
but the Nix/NixOS installation command used is the slow one. Nix doesn't have
versions for packages in the traditional sense. A fairer pair of test
commands:

    
    
      nix-env -iA nixpkgs.ack
      nix-env -iA nixpkgs.qemu
    

This is because Nix considers an entire tree of Nix expressions at once,
rather than examining each package in isolation. When nix-env is run without
the -A flag, it must evaluate the entirety of nixpkgs and then search for the
desired package amongst the complete tree. With -A instead, it is possible to
evaluate only the portions of the tree which are relevant to the named
package.

~~~
tln
Yeah, this is much speedier!

On my machine, for the original command I get:

    
    
       created 56 symlinks in user environment
       real 0m 25.64s
    

For your command I get:

    
    
       created 56 symlinks in user environment
       real 0m 4.39s

------
mfer
Context matters. There are some underlying assumptions not discussed here. You
can see them in the content like...

> Pain point: too much metadata

and

> I expect any modern Linux distribution to only transfer absolutely required
> data to complete my task.

Consider the case of searching for a package. That search can be performed
locally and you would need the metadata locally or it can be performed
remotely and the results sent over the wire. It has to be performed somewhere.

If performed remotely than the remote endpoint knows what you've been
searching for. If you are trying to perform a search across multiple package
repos how do you to that securely? How do you not leak information? The remote
endpoint can capture, log, and use information about who searched for what.
Then tie that back to company and other data. There's a lot of sensitive
corporate information that can be contained in these searches. So, do you do
perform searches remotely or locally? Many systems choose locally but that
entails downloads a dataset.

There are a bunch of things like this to take into account.

Other things, like serial downloads, are likely a product of the package
managers age. I would hope modern ones work out parallel downloads.

~~~
choeger
You both have a point here. As a long time fedora user and occasional packager
I always considered the sudden meta data downloads a terrible issue. On a slow
connection not would basically disable your package manager due to timeouts.

I never looked into the technical details of yum/dnf metadata, but I suspect
there must be a much smarter way to manage that stuff. Incremental updates for
instance, or maybe just a different format (is it not xml?).

------
openfuture
People are asking 'why does speed matter?'

If you use nix then you'll know that a “nix-shell -p” is basically the url-bar
of the browser.

If a browser is a VM in your OS then why not run distri in a VM and speedily
launch native apps, horribly useless example.. for now.

If the browser is eating the world anyway then incentives to have native apps
be incompatible between operating systems fade away and throwaway envs like
nix-shell (and to a lesser extent nixos-shell) become even more interesting. .
So long as they are fast enough to feel as effortless as opening a web page.

------
kbenson
The author combined fetch and install, which is just silly. Fetch times are
affected by any number of things, such as the number of mirrors the package
repos have, whether you have a mirror near you, whether your system is
configured to use it, and your current connection speed.

Both fetch and install are useful to know, but for very different reasons.
Only so much of fetch speed is in control of the distro setting it up (but you
can set up your own mirror if you like...), but install speed _once the files
are already staged locally_ is also useful to know. Additionally, whether the
package manager keeps a local cache of metadata for packages, and whether it
checks and updated it every request or only at certain intervals or when
requested will affect this as well (and is _very_ important, if you want to
know about security updates immediately and not after some interval of hours).

What the author has done is the equivalent to testing how fast certain one
mile and 10 mile stretches of highway are, but without controlling for time of
day, and not realizing rush hour is invalidating all the results. The first
step to measuring something is to learn about what you're measuring so you can
avoid issues like these.

------
nemetroid
Arch has a much longer dependencies list for qemu than Alpine does[0, 1]. Arch
has 7 sub-packages to qemu, while Alpine has 82. This suggests to me that
Arch's base package includes functionality that Alpine has placed into
subpackages. Not all packages named "qemu" are created equal.

Also, without separating out the time spent waiting for the download, the
author's choice of mirrors probably has a significant impact.

0:
[https://www.archlinux.org/packages/extra/x86_64/qemu/](https://www.archlinux.org/packages/extra/x86_64/qemu/)

1:
[https://pkgs.alpinelinux.org/package/edge/main/x86_64/qemu](https://pkgs.alpinelinux.org/package/edge/main/x86_64/qemu)

------
4ndrewl
For me, the thing that's missing from the article is the 'so what?'

If Linux package managers were faster then <it would affect me how?>

~~~
rwmj
I'd be happier with not having to wait so long for dnf metadata to download
all the time. Pretty much my job (at Red Hat!) is to wait for it a lot of the
time.

~~~
curt15
Isn't the main reason dnf metadata is so massive that it contains the
filelists of every package in repository? No other package manager requires
that information to work. Does anyone else at Red Hat (such as the dnf
developers) consider the metadata size a major problem?

~~~
rwmj
The filelists are a good thing since they let you do:

    
    
       dnf install /usr/share/mypackage/file
    

However they are _not_ downloaded if you're just installing a package by name
(even for dependencies) which is what this test is all about.

~~~
curt15
Then what constitutes the bulk of the metadata? I seem to recall it being much
smaller back in the days of yum (~20mb), and the number of packages in the
repository has not ballooned that much since then.

------
riscy
The main use cases I can think of for a super fast and lean package manager
are (1) speeding up docker image builds (only matters when new dependencies
are added), (2) folks with slow / significantly metered internet connections,
and (3) embedded systems / low resource machines.

Otherwise 20-60 seconds saved doesn't matter to me personally for a one-time
event. I'm guessing it doesn't matter to the package-manager developers
either.

~~~
groundlogic
Always strive for excellence. For something as fundamental as this, it's a bit
shameful that mainstream package managers are so unnecessarily slow.

Let's say there are 30 million Linux users who are wasting an hour every year
on slow package managers. 30 million hours means 3424 years wasted every year
or the total average lifespan of 48 people. Package managers kill 48 people
per year. ( /s )

(There's some related Steve Jobs quote here somewhere.)

------
groundlogic
Like the article notes, the Alpine Linux package manager (apk) is refreshingly
fast.

It's odd that Alpine lets you install e.g. "emacs-nox" on an IoT device
connected via crappy LTE, on the other side of the world, faster than you can
install it on your local Ubuntu dev machine, with like 100-1000x more network
bandwidth, 10-100x more CPU and 100-1000x faster local storage.

------
techntoke
Based on these benchmarks, you are including the time it takes to download
packages, which is very unreliable since it varies widely based on mirrors.
I've installed QEMU on Arch before in a much shorter time. Can confirm though
that Alpine has very consistent download times, while sometimes with Arch I
will have to change my mirror due to poor speed.

~~~
aruggirello
I'm using a local apt-cacher-ng to save bandwidth, and apt really flies - I'd
recommended it if you have (obviously) 2+ (debian, Ubuntu or derivatives)
machines. Makes even the occasional network slowdown much more tolerable...
Just disable it when doing dist-upgrade or do-release-upgrade.

------
wwarner
I think to go really fast, you'll have to introduce a new data structure,
similar to git's. (I'm thinking it's a trie, but I'm coming up zero evidence
for that.) Calculating the dependency graph could be nearly instantaneous with
an improved data structure on the client. Syncing the local tree with the
remote authority can also be blazingly fast. After that, if you really want to
be smart, you can use an rsync style transfer that only copies differences.
And if you want to boil the whole ocean, make all the package maintainers
conform to a standard that reduces installation to a pure copy operation.

It'd be awesome if updating a node was a sub-second operation, it's definitely
a big benefit to cloud based computing.

------
catern
There's another flaw besides the many flaws that others have mentioned. As the
author says, the amount of metadata downloaded is different between package
managers. Perhaps ideally, only the metadata needed for a requested package
would be downloaded. But I don't believe that's what's happening.

Alpine is fastest not just because it has a more efficient metadata format,
but also because it has far fewer packages than the others.

------
bryanlarsen
For some reason (probably transparency), rather than using specialized docker
images the GitLab auto-devops scripts install needed packages into a generic
container. This really slows down the CI pipelines and is super painful

At least the ones where the generic container is Ubuntu/Debian based. `apt
update && apt install -y` is really slow. But alpine based containers with the
equivalent `apk install` are blazing fast.

~~~
dgruesso
GitLab PM here. Auto DevOps makes use of herokuish in order to find the
language of your project. Herokuish images are Ubuntu-based, and it doesn't
allow the use of specialized images. One can always provide a dockerfile
(based on the image of your choice) and Auto DevOps will use that instead.

------
teilo
While a comparison like this may be interesting trivia, it's irrelevant for
all practical purposes.

Consider the amount of time it takes to update a system vs. its overall
runtime. Now consider what's important in a package manager. Consistency,
rollback, configuration management, audit trails, etc. The speed of the
package manager is at the absolute bottom of things to be concerned about.

~~~
_frkl
Maybe for you, but there are use-cases and situations where this adds up, and
has monetary effects. Container builds, or CI pipelines that run often would
be one example. Sure, the things you mention are important, but all other
things being equal, everybody would prefer faster installs to slower ones,
right? So it's a fair thing to worry about, me thinks.

------
zx2c4
Meanwhile Gentoo's portage took almost 7 minutes to build qemu...

    
    
        zx2c4@thinkpad ~ $ genlop -t qemu
             Sat May 18 15:36:44 2019 >>> app-emulation/qemu-4.0.0-r2
               merge time: 6 minutes and 49 seconds.

------
IloveHN84
Still faster than Windows and Apple updates

