
Privatization Means Rebuilds - protomyth
https://www.dragonflydigest.com/2016/01/27/17547.html
======
ChuckMcM
Cached version :
[http://webcache.googleusercontent.com/search?q=cache:_t5yQSL...](http://webcache.googleusercontent.com/search?q=cache:_t5yQSLC0xIJ:https://www.dragonflydigest.com/2016/01/27/17547.html&num=1&hl=en&gl=us&strip=1&vwsrc=0)

And the summary is, separate "system" requirements/dependencies from those
that packages depend on so that rebuilding a package doesn't break the system.

~~~
protomyth
I think it is the opposite way around, so that rebuilding the system doesn't
break a package.

Basically, they can switch system libraries or delete them entirely without
the packages being affected since they will be using the non-private libraries
from packages / ports.

------
SwellJoe
I still can't understand why several BSDs so stubbornly cling to ports as a
"solution" to package management. I ran into this problem, and variants of it,
over and over again when trying to support FreeBSD in our products, in the
past. Though pkgng may resolve some of my problems with package management on
FreeBSD (it at least allows third party repositories, the lack of which was a
big pain point for me in the past), I haven't tried it yet, and from what I
can tell FreeBSD still favors ports, and several other BSDs are still using
ports exclusively.

I've even seen this library mismatch happen on a couple of occasions in a
fresh install, where installing a variety of ports leads to one or some of
them being broken immediately after install because several packages called
for different versions of libraries. And, the longer a BSD system is in
service, the more likely some portupgrade, or installation of a new package,
is to break some other thing on the system, requiring far reaching rebuilds
(and sometimes, manual intervention, because the dependencies specified aren't
sufficient to allow the system to automatically fix itself).

I begin to feel like a broken record saying this again and again, but: ports
is a clever tool, but it is not a solution to the problem of package
management.

There's so much to love about the BSDs, but lack of effective package
management has always made it a non-starter for any purpose, for me. Effective
package management is not esoteric or bleeding edge or experimental stuff, at
this point. deb-based systems had apt 20+ years ago, RPM based systems had yum
~12 years ago. Slapping a band aid on the darned thing doesn't fix it.

And, so many conversations about packages devolve into ports proponents
missing the point, and saying, "Well, you can install binary versions of
ports, so it's fine." When the availability of binary packages is utterly
orthogonal to the problems I'm talking about (though also a big deal, and
building from source on a production system is silly and potentially
dangerous).

So, I think I need to stop complaining about it, and start asking the right
questions of BSD users. Why is ports such a fiercely defended thing when it
has so many weaknesses compared to modern package managers? Does pkgng
actually solve these problems (I can't really tell, it seems to be a meta-
package system that still sits atop ports)?

~~~
marios
OpenBSD has both binary packages and ports. Actually, the packages are built
from ports, so if you want to customize a package, you checkout the ports
tree, adjust the Makefile, add patches or whatever it is you need, and then
you get a nice package that you can use on other OpenBSD installations
(provided they are running the same version).

This is similar to what one does with, say, APT (apt-get {build-dep, source}
<pkg>; modify; dpkg-builpackage ...).

When you get install a release, you set a package source (/etc/pkg.conf is
more or less the equivalent of /etc/apt/*), and then you just pkg_add <pkg>.
Package versions are fixed, pretty much the same way you get when you run
debian stable.

There is one major difference though: OpenBSD is an OS, while Debian is a
distribution. This means that debian will install the package files relatively
to /, while OpenBSD will install them relatively to /usr/local.

It may sound stupid if you've only used Linux before, but it makes sense:
files in /usr/bin, /usr/sbin,... are files that were shipped with the OS. What
lives in /usr/local is what you installed on top of it.

Other than that, I don't see any major difference. You occasionally get some
libraries that are out of sync with a package, but that happens on -current
and it's usually that the package was not rebuilt yet on the mirror you're
using. (Ever run Debian sid ? Well, it's the same thing)

What I'm saying is that OpenBSD uses ports effectively and in day to day usage
I see no difference to APT-based distributions. I've never used it, but I hear
very good things about pkgsrc (developed by NetBSD but it can also be used
elsewhere. AFAIK, Joyent uses it and some people run it on Linux too). From
what I can see on the homepage, it provides both sources and binaries.

Until pkgng was introduced, FreeBSD was lagging behind but from what I can
tell, it seems the situation has improved.

What "weaknesses" related to ports do you have in mind ?

~~~
SwellJoe
"What "weaknesses" related to ports do you have in mind ?"

The ones I've mentioned (and mentioned in the article we're commenting on):

\- Library mismatches leading to broken packages, sometimes on a freshly
installed system. I've always heard outright denial that this happens, but
I've literally never installed a FreeBSD system that _didn 't_ end up with
something broken by the time everything needed for a fully functioning web
hosting system was installed. It's like there's Stockholm syndrome happening,
or something.

\- Poor dependency resolution, and weak validation tools, that requires manual
intervention to prevent or repair those breakages which are bound to occur.

Other complaints I have about ports:

\- No reasonable mechanism for offering third party package repositories.

\- Default ports are usually anemic, missing very basic functionality,
guaranteeing you have to build some things from source in order to get a
really functional version of, say, Apache. Now, you have to make your own
means of distributing these custom builds across your systems, _without_ any
reasonable means of setting up your own repository (you can duplicate the
whole ports tree, and merge your changes, which is ridiculous, and the tools
to help with this are weak).

Really, the biggest failings are in dependency specification and resolution. I
don't know if it is a process failure (e.g., nobody is taking the time to
specify things precisely enough to end up with a working package in every
circumstance) or failure of the system (e.g., the system doesn't provide the
means to specify things precisely enough to insure a functional system), or a
combination of the two. But, the end result is that it is frightfully easy to
end up with broken software, and often requires a lot of knowledge and human
interaction to fix it.

Again, providing binaries is fine and all (and, I think, mandatory for large
automated deployments), but is not the basis of the problems I've experienced
with ports. The fragility of ports is my problem with it, not the surface
level differences between how things are installed and updated.

~~~
tw04
>\- Library mismatches leading to broken packages, sometimes on a freshly
installed system. I've always heard outright denial that this happens, but
I've literally never installed a FreeBSD system that didn't end up with
something broken by the time everything needed for a fully functioning web
hosting system was installed. It's like there's Stockholm syndrome happening,
or something.

I'm not questioning that is your experience, but I am curious what packages
specifically, and what specifically broke. Because in my almost 20 years of
using FreeBSD, I have literally never once had that happen.

Just generically saying "It always breaks" without examples is pretty
difficult for anyone to defend or even begin to try to counter.

~~~
SwellJoe
Some that come to mind:

On FreeBSD 8 (I think), installing Postfix alongside several other packages
that relied on OpenSSL led to Postfix breaking, because the OpenSSL version
was updated but Postfix was not. This happened regardless of binary or source
install. I don't remember the specifics, honestly, as it's been a while. But,
it required manual intervention and patching the port to make it rebuild a
functioning Postfix.

On some earlier FreeBSD version, I had a dependency hell situation arise with
Apache. Again, some library got moved out from under it, and dependencies
weren't sufficiently specific to allow it to be automatically fixed, even with
a complete rebuild of all of the packages (which would be a ridiculous
requirement that I would still complain about..but even that wouldn't work).

On the same system, saslauthd was utterly broken for months (maybe even
years), whether installed from binary or source (there was an issue about it,
nothing happened). It required me to manually patch it and update it to make
it work.

Also, despite assertions to the contrary, I've found that "mixed" systems of
binaries plus ports also often leads to brokenness (the port install will lead
to a library being upgraded out from under the binary install software,
requiring a port install from source to fix the problem). That should never
happen. Random shit should never just stop working because you installed
another package, and I've found it to happen at least once or twice on any
FreeBSD system I've ever dealt with.

All of this is fuzzy memories. It's been a few years since I earnestly tried
to support FreeBSD in our products, because the package management situation
was such an ugly mess. Before that some of the kernel limits were problematic
(like group membership was limited to 128 or 256 groups, which broke our
permissions model for Apache back then, so we had to rebuild the kernel just
to make our software functional). There's just been a lot of pain involved in
making FreeBSD work for us, and I've spent more fruitless hours trying to make
it work than any other system we support (and for a much smaller pool of
paying customers...we can count the FreeBSD users on our fingers, vs. a few
thousand Linux users). Despite a _very_ low return on investment, I still dig
in every few years and waste hours of my time fighting with shitty package
management, because I really want to support it.

Anyway, let me be clear about this: If you have to manually patch or edit
_anything_ to get a functional installation, your package management tools are
broken. In my experience, that's always been required. Admittedly, our stack
of dependencies for our software is very large (Apache, MySQL, PostgreSQL,
Postfix, Dovecot, Cyrus saslauthd, BIND, procmail, spamassassin, ClamAV, Perl
and several modules, bash, awstats, modern Python and Ruby versions, and more
that isn't coming to mind right now).

Can I make it work? Yes, I've done it a handful of times in our ten years of
making Virtualmin as a commercial product. I honestly want to support FreeBSD,
and always have wanted to support FreeBSD. xBSD users are almost universally
extremely competent; they file useful bug reports, they're helpful when I have
problems based on the limits of my knowledge of FreeBSD. But, it's always a
massive pain in the ass, and when a new version rolls out, I find I have
another stack of problems to work out manually. Most of those problems are
related to package management failures.

Also, while xBSD users are often awesomely bright folks, they're also
extremely defensive, to the point of it being difficult to address problems
(like this one). We're talking about a post in the DragonflyBSD list that
discusses _this very problem_ , the exact thing I'm belly aching about, is a
thing that is acknowledged as a longstanding issue in ports by the developers
who have made some changes to try to address the problem. Why are we arguing
about whether it happens? (I understand DragonflyBSD and FreeBSD have diverged
many years ago, and there are significant differences, including in package
management, but this seems to be a problem of ports in general, in my
experience.)

~~~
tw04
>(Apache, MySQL, PostgreSQL, Postfix, Dovecot, Cyrus saslauthd, BIND,
procmail, spamassassin, ClamAV, Perl and several modules, bash, awstats,
modern Python and Ruby versions, and more that isn't coming to mind right
now).

Outside of Postgres AND MySQL on the same box... your requirements are
actually pretty standard, and something I always deploy. I guess I'll leave it
to others to comment - but that's basically my standard install, so I'm at a
loss. I've never had it break, thought perhaps you had some odd corner-case
package that wasn't properly being vetted out. Unfortunately we can't rewind
time - I'd strongly suggest you post the bugs on the bug tracker in the
future, because your experience is definitely not something I'd consider
normal.

>We're talking about a post in the DragonflyBSD list that discusses this very
problem, the exact thing I'm belly aching about, is a thing that is
acknowledged as a longstanding issue in ports by the developers who have made
some changes to try to address the problem. Why are we arguing about whether
it happens?

Why are we arguing about whether it happens? Because you didn't report it.
When I asked for specifics, you should've provided a link to the FreeBSD bug
tracker where you reported the issue. Instead, we're talking about it on a
social media site 7 years after the fact, with vague details about the
packages that caused the issue.

I literally don't have a horse in this race (I don't contribute to FreeBSD, I
don't rely on it for my livelihood, I'm literally just a longtime luser) - but
it honestly feels like you're attacking the project with unsubstantiated
claims.

~~~
SwellJoe
_" Why are we arguing about whether it happens? Because you didn't report
it."_

On what do you base that assertion? (Actually, I didn't need to report most of
the problems I ran into. They were already reported, but not fixed, when I
came upon them. But, I've been an open source developer for 20+ years. I
report shit.)

------
stock_toaster
This is great. A good example of why this is great, is openssl. Because
ports/packages build with the base openssl, a security team supporting a
release usually has to maintain an abi compatible version in base. If
ports/packages simply used their a packaged of openssl, it would make it much
easier for ports/packages to update in lockstep, as well as the base system to
update on its own schedule (update to newer version and simply update any
outdated api usages in base as well -- and for example to use libressl
instead).

I think BSD's exporting so many libraries as part of the base install made
much more sense in the time with small hard drives, but maybe not as much
today.

~~~
xenophonf
The following isn't exactly right:

    
    
      Because ports/packages build with the base openssl,
      a security team supporting a release usually has to
      maintain an api compatible version in base.
    

Forcing ports to compile against the security/openssl port instead of the base
version is pretty easy. You add the following knob to /etc/make.conf, to the
port's Makefile, or to the make(1) command line:

    
    
      WITH_OPENSSL_PORT=yes
    

The reason the security team has to maintain an API/ABI-compatible release of
OpenSSL is because FreeBSD releases make that guarantee for all base system
libraries---and the reason OpenSSL is in the base system in the first place is
because other things in base like OpenSSH and Heimdal require it. This
guarantee lets me build packages on FreeBSD 10.0 that work fine on FreeBSD
10.2 (and, generally, vice versa).

~~~
stock_toaster

      > Forcing ports to compile against the security/openssl port
      > instead of the base version is pretty easy. 
    

Yes. I do this currently, and it works great. But not everyone does, so the
security team _has_ to keep abi compatibility.

    
    
      > The reason the security team has to maintain an API/ABI-compatible release
      > of OpenSSL is because FreeBSD releases make that guarantee for all base
      > system libraries---and the reason OpenSSL is in the base system in the
      > first place is because other things in base like OpenSSH and Heimdal
      > require it.
    

_Head scratching_ That was...kind of my whole point. Anyway, openssl is
planned to become private, so I just need to wait a bit (maybe Allan will get
it into FreeBSD-11? see[1]).

[1]:
[https://wiki.freebsd.org/OpenSSL/Base](https://wiki.freebsd.org/OpenSSL/Base)

------
Animats
I didn't know that BSD now had distros.

~~~
2trill2spill
It's not a distro, dragonflyBSD is it's own Operating system. DragonflyBSD
split from FreeBSD when they disagreed on how to do SMP.

But I guess PC-BSD could be considered a distro.

~~~
groovy2shoes
The "D" in "BSD" stands for "distribution"...

