
Building a “real” Linux distro - ddevault
https://drewdevault.com/2017/05/05/Building-a-real-Linux-distro.html
======
tanderson92
One can bootstrap a system with a source-based package manager too. Here[0] is
the process for bootstrapping a _new architecture_ on Exherbo Linux. Note that
the package manager is built on the host system and installs to the new
system.

Exherbo supports cross-compiling with multiple concurrent architectures (both
called 'multiarch') and installing into non-host environments.

A lot of other cool stuff, but the biggest point is that one can do what this
blog describes with dependency management taken care of. The other point is
that simply by managing the special 'system' set, one can pare down the base
system. The OPTIONS flag can reduce system size by minimizing unneeded
features (similar to Gentoo's USE).

[0]:
[http://exherbo.org/docs/bootstrapping.html](http://exherbo.org/docs/bootstrapping.html)

------
peterwwillis
Can somebody explain to me the obsession with static linking? Do these people
not realize that performance gains from shared memory between disparate
applications hinges on dynamic linking mmap'd libraries and COWing shared
memory sections to private on demand, reduces duplicate code, standardizes
operations, and simplifies debugging? Has the world gone mad?!

~~~
Sir_Cmpwn
Most of the applications you will have running at any given moment probably
have few libraries in common. A typical linker will only include in the final
executable any functions you require from your dependency as well, which can
dramatically reduce the footprint of a piece of software (when considered in
combination with its dependencies). Statically linked binaries also load
faster and don't break when you have a botched update.

My distro does support dynamic linking, but prefers static. In practice the
RAM footprint and disk footprint is much, much smaller than any dynamically
linked distro I've tried. Avoiding GNU also helps a lot to reduce bloat.

Static linking also makes agunix more portable. Stuff like this is possible:
[https://sr.ht/5HTs.png](https://sr.ht/5HTs.png)

~~~
peterwwillis
I have 878 unique libraries shared between currently running processes on my
Linux laptop at this moment, with the amount of processes sharing a library
ranging from 2 processes to 820. Most may have few in common, but there are a
lot of them, and the mean is 62, mode is 4.

As far as the botched update argument: When does anybody botch an update of a
shared library? When they're forcing something to install, or they don't know
what they're doing. On the other hand, mis-matches in applications using
different versions of the same code causes silent, unexpected, difficult to
debug errors.

I'd also be interested in seeing actual "RAM and disk footprint" numbers as a
comparison for the same applications and libraries, because that doesn't
really make sense. Even if applications only copy a couple small functions
each, they add up, whereas the shared library only has one copy.

I can't speak to the speed differences (again, I would like to see numbers)
but certainly different versions could lead to competing cache and code path
hits.

And a static application has nothing to do with portability. Just because I
can run a Windows 95 application in Windows 10 doesn't mean it's going to work
right, and just because FreeBSD has a Linux binary compatibility layer doesn't
mean the app is going to work. Portability requires making it portable first.
You can copy shared libraries with a dynamic executable and run it as reliably
as a static app.

~~~
Sir_Cmpwn
>I have 878 unique shared libraries shared between currently running processes
on my Linux laptop at this moment, with the amount of processes sharing a
library ranging from 2 processes to 820. Most may have few in common, but
there are a lot of them, and the mean is 62, mode is 4.

Good god, this tells half the story. An agunix install booted to a shell will
typically only have 3 processes running: runit, a getty, and your shell. I
don't think I've _ever_ had more than 50 processes running on my agunix system
at any given moment.

>As far as the botched update argument: When does anybody botch an update of a
shared library? When they're forcing something to install, or they don't know
what they're doing.

For any number of reasons. This has happened to me dozens of times in the past
year. You also run into issues with any software you compiled yourself that
aren't managed by the package manager (or at least not supported by your
distro). You know what's better? An executable that works on any Linux system
and will continue to work for the next 20 years without any changes. How about
running two executables that need different versions of a library?

>On the other hand, mis-matches in applications using different versions of
the same code causes silent, unexpected, difficult to debug errors.

I haven't run into this yet. Have you?

>I'd also be interested in seeing actual "RAM and disk footprint" numbers as a
comparison for the same applications and libraries, because that doesn't
really make sense. Even if applications only copy a couple small functions
each, they add up, whereas the shared library only has one copy.

My laptop is currently running runit, 8 gettys, sshd, wpa_supplicant, crond,
udevd, and dhcpcd. It's using 60 MiB of RAM. So far as disk footprint is
concerned, see package sizes at
[https://repo.agunix.org/](https://repo.agunix.org/). Base system is ~350 MiB
on disk and there's still room for improvement.

From the agunix home page:

>Our first concern is to build a lightweight POSIX environment - a familiar
design for users familiar with other Linux distributions is not our goal.

It's better to think of this as a new operating system entirely than as a new
Linux distro.

~~~
peterwwillis
It sounds like you're running a hobby Linux distribution. I have nothing
against that, but please don't make it seem like somehow it is superior just
because it is different. I used to make Linux distributions that could fit on
a floppy disk, but I wouldn't brag about how fantastically small it is and how
that obviously runs much better.

If you are compiling source code by hand, you are circumventing the entire
distribution. Do not expect it to work. Just because it may work a lot of the
time for you does not mean you should expect it to. If updating a shared
library broke something, it's because you didn't know what you were doing.
This does not mean shared libraries are bad. In fact, a great feature of
shared libraries is the ability to update thousands of applications by simply
changing one object file. LD_PRELOAD and many other features are incredibly
handy, and don't work with statically compiled apps.

Yes, I have run into dependency incompatibility bugs. It happens when expected
behavior in library code changes. If an application was not written expecting
that behavior, then when it interacts with another application, it may have a
conflict. The common places it will crop up are IPC, protocol, and file format
changes, but also locking, support for optional functions, infrequently used
features, etc. It can also happen from updating software's configuration in a
way that is incompatible with some older version of some software that was
using it, even though some other software supports the newer functionality. If
you were around during the Linux desktop wars this was common. It happens less
often with distributed applications since they mostly use APIs now, but some
APIs suck balls and don't have versioning, and people will change the
functionality of some part of the API that old code was depending on
somewhere.

Copy an application from 5 years ago into a new system, and copy all the
shared libraries into the new system too, and it should run, just like a
static binary would run. And it will then immediately die, because all the
other dependencies that don't have anything to do with linking weren't there,
like kernel feature or modules changes, missing or changed non-library files,
protocol changes, missing/incompatible daemons, abandoned deps, security
paradigm shifts, etc etc.

Honestly, tracking and updating multiple copies of software with different
dependencies is not difficult. We were doing that a decade ago at scale (tens
of thousands of servers, thousands of individual software packages). People
just don't like dealing with reality. So instead of dealing with the fact that
they are doing complex things, they pick up their mess and throw it into a
spare room. Look, the living room is clean now! Let's hope nobody looks in the
other room.

------
fhood
Could someone elaborate on the "cons" the article mentions? I don't totally
understand why those would be problems.

~~~
Sir_Cmpwn
I originally went into more depth, but replaced it with a small summary of my
motivations. The article is more about making a distro than about agunix
itself. See the website for some information about agunix.

------
jacobroyquebec
What's so wrong about GNU?

~~~
thesuitonym
This is really a case of "If you have to ask, you'll never know."

That's not meant to be derogatory, just that if you don't have a problem with
GNU, you _won 't_ have a problem with GNU, because there's nothing wrong with
it, it's jut a matter of opinion (usually about the GPL).

~~~
jacobroyquebec
Allright! The article nor the FAQ page didn't mention why it was hated so much
haha.

Thank for the answer!

~~~
Sir_Cmpwn
I updated the FQA with some stuff discussed in these comments:
[https://agunix.org/fqa](https://agunix.org/fqa)

~~~
jacobroyquebec
Thanks! BTW, It seems to be a really tedious work, I like the result :)

------
eatonphil
This is really cool, Drew. I can get behind a lot of the motivations here. I
didn't realize clang can't compile Linux. Doas is awesome on OpenBSD and I've
been using its port on FreeBSD too. I'm glad to see the Linux port getting
more use too.

The coolest part about NetBSD for me right now is that you can cross-compile
it on other OSs. That's quite a feat. However, OpenBSD has my heart for
simplicity of core code. It has become my go-to reference for generic userland
and kernel questions.

I'll have to follow the guide for installing this sometime. Some great stuff
to learn here. Overall, keep up the awesome work. Seriously.

------
psadauskas
My 2012 MBP is getting a bit long in the tooth, and had several software-ish
issues that were totally unfixable. As a long time Linux user, I decided to
switch back, and purchased a powerful laptop to replace it. I did a similar
survey of distros, and haven't been happy with any of them.

Ubuntu worked fairly well in a vanilla configuration, but as a developer, I
prefer a more powerful and customizable desktop UI. My preferred window
manager is awesomewm, but trying to get it to integrate with Unity or Gnome-
shell is pretty much impossible. If I run just bare awesome, I lose all the
power management & audio niceties. I can probably set it all back up, but
documentation on how to do so is slim. Additionally, installing newer software
than what is provided by apt is a giant pain. You basically have to hope the
vendor provides their own apt repository for you to use, and that they've
updated it for Zesty.

Arch linux was by far my favorite before I switched to OSX ~10 years ago. It
always has the latest software available in core. Additionally, AUR provides
betas and even more esoteric software. Anything I come across here or on
reddit probably already has an AUR package for it, and in the rare case it
does not, it is trivial to write a PKGBUILD for it. However, installing Arch
is way harder than it used to be, particularly on laptops. There's so much
variety and complexity in hardware now, you could spend a month getting
everything set up and it still would not be satisfactory. Power management,
sound, full-disk-encryption, webcam, display hot-plugging, hardware sensors...

I used Fedora for quite awhile on a desktop before I got this laptop, and
really enjoyed it. However, the latest Fedora uses wayland for everything,
which doesn't work with the closed nvidia drivers, and nouveau on the 1070 in
this laptop basically doesn't work. However, `dnf` is 10x better than `apt`,
and I found it much easier to find copr repositories for updated software.
Maybe I'll revisit once wayland and nvidia work together.

I wish there was a distro half-way in between Arch and Ubuntu/Fedora. A nice
installer that autodetects and sets up hardware automatically, but still easy
to hack on. The problem with the former, though, is you pretty much need the
resources of Canonical or Red Hat to get it done.

Also, for the record, I love systemd. I've been using unixes for 20 years, and
never managed to grok init scripts that I was comfortable creating or
modifying them. Systemd (like Upstart and Launchd before it) is a huge
improvement over that mess.

------
shmerl
Cons include ... systemd. The author doesn't elaborate how it's a con, same as
some other examples.

~~~
RX14
It's clear the author thinks it's a con, but I don't see why this article
should justify that, it's a pretty unrelated topic to the one at hand.

~~~
shmerl
The tone of the article suggests it's something obvious and universal, which
it's not. So there should have been some elaboration, or at least something
like "for my purposes".

~~~
Sir_Cmpwn
[https://agunix.org/fqa](https://agunix.org/fqa)

~~~
shmerl
Most of that sounds quite debatable and a matter of taste. I don't find those
arguments objective enough. Calling anything they don't like "bloated" doesn't
sound right.

~~~
Sir_Cmpwn
Have you ever dug into the internals of GNU tools and compared them to other
tools in the same niche? It's self-evident.

------
davexunit
Yet another anti-GNU, anti-systemd article.

For folks interested in reading about how distros bootstrap themselves, the
GNU Guix manual has some good details:

[https://www.gnu.org/software/guix/manual/html_node/Bootstrap...](https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html)

~~~
Sir_Cmpwn
No, it's an article about making a new Linux distro, with a brief summary of
my motivations. Someday I might write an anti-GNU, anti-systemd article, but
this is hardly it.

~~~
davexunit
If you are anti-GNU and anti-systemd, I am highly skeptical of your design
choices.

