

Building packages at scale - jperkin
http://www.perkin.org.uk/posts/building-packages-at-scale.html

======
snw
Interesting article with historical context that goes 10 years back and many
detailed insights!

Using pkgsrc at our company since a bit over a year has been excellent thanks
to the great work from jperkin and many others. It is one of the tools i wish
i had known earlier about since it makes life sooo much easier.

\- Quartly stable releases mean that you have recent software most of the
time.

\- It is available on many platforms (OS X, Linux, SmartOS/Illumos, the BSDs,
...) so you can use the same packages on all your systems.

\- Very easy to build binary packages of custom software / custom options /
different versions / add your own patches. This is critical for most
sysadmins/ops people - and if you ever tried to setup koji for CentOS you will
love the simplicity of pkgsrc...

In my opinion pkgsrc is the most underappreciated "devops"-tool out there. It
gave us much more control over our software stack and reduced the amount of
work we had to put in to archive our goals.

I also expect the effort to reduce bulk-build times to further improve the
software quality of all things in pkgsrc. Instead of weekly/daily bulk-builds
(depending on the platform) it will be possible to have 4 or more reports a
day. This quick feedback will make determining if/which commit broke/fixed
something more immediate. Great stuff!

------
kardos
Great article, impressive attention to detail. It sounds like a situation that
would benefit from a compiler cache, such as the one found at
[https://ccache.samba.org/](https://ccache.samba.org/). Is there any benefit
to be had there or is the storage tradeoff too expensive?

~~~
jperkin
I want to try ccache at some point (with pkgsrc it's trivial to add), but I'm
not convinced it will help.

Given the distributed nature the first implementation would keep the ccache
objects on NFS, and that's going to cause additional latency for every cc
invocation to the point where even a cache hit will likely be slower.

The next step would be to synchronise all of the ccache objects back to each
build zone and then loopback-mount the directory into each chroot. This may
provide a small increase in performance, but at the cost of additional
complexity. I also wonder whether it would be enough of a performance win
compared to the extra time it will take to rsync the objects to each build
zone at the start of the build.

I will likely do this at some point anyway, as I'd like to have some hard
numbers to back up my theory, and if it turns out to be a win, even better ;)

~~~
tlb
I've built a few such configurations myself, but never again. The worst
behavior is that sometimes, the NFS caches result in compiling old versions of
some files some of the time. When you're rapidly tweaking things for debugging
and sometimes some changes only make it into some compilation units, it can
drive you insane.

And it doesn't go much faster than just compiling locally with Clang on an 8
core CPUs with SSDs.

