
I Was Wrong about Nix - xena
https://christine.website/blog/i-was-wrong-about-nix-2020-02-10
======
nitrix
As much as I admire the people involved with Nix, taking the initiative to
solve what they believe can be improved and the other developers like her,
toying with new things and successfully managing to figure all this stuff
out... I'm actually disappointed.

The thing is, Nix is an absurdly complex piece of software, perhaps on a
similar order of magnitude as maybe Kubernetes or other gizmos, requiring a
very large time commitment to understand and to use properly.

Just for fun, have any of you tried to keep up during the Docker bigbang, with
all these technologies popping up left and right, switching names, dying
abruptly, getting super-seeded or hacked with vulnerabilities? That was one of
my most mentally draining year for me and I've been in the field for a while
now.

See... along the way, I've learned that Computer Science is always about
tradeoffs. Now when I'm being asked to trade away my sanity learning this
nonsense for in return a couple of megabyte of disk space that could've been
wasted, I just don't see the value.

Seriously, are we optimizing for the right stuff here? I know Nix is doing
package isolation with a programmable environment defined by text files, it's
immutable and all. I get it. But we have alternatives for these problems. They
do a pretty good job overall and we understand them.

Ironically, even those solutions I'm comfortable with are still struggling to
get good adoption! How the heck is our field going to avoid fragmentation when
it keep growing exponentially like this.

Perhaps it's just me aging and getting groggy; though there must be an
explanation for this phenomenon. Please, it definitely gnaws at me.

~~~
Aloha
We just keep trying to hide complexity and fragility with more layers of
redirection and abstraction - it doesnt solve the problem, but it makes it
tomorrows problem.

In the end, excessively clever people make me nervous, because they leave
complex problems in their wake, which I'm not clever enough by half to solve.

~~~
patrec
This is an annoyingly ignorant comment.

Nix, unlike say docker, actually solves a problem correctly in a principled
way: reliably and reproducibly going from source artefacts and their
dependencies to installed binaries/libraries/etc and without conflicts or
undesired interactions between other installed binaries/libraries/etc, even
different versions of the same thing.

It represents a major innovation and step forward in the depressingly
dysfunctional mainstream OS landscape, that still is stuck with all the
abstractive power of 80's BASIC but with concurrency and several orders of
magnitude more state to PEEK and POKE.

~~~
pjmlp
It doesn't solve it, because it is confined to a relatively niche GNU/Linux
distribution, unlike Docker.

~~~
patrec
Nonsense. You can run nix on basically any linux system (not just NixOS) and
macOS. Not just theoretically, I'm doing both and am not running NixOS itself
at all ATM. It's also the only sane way to build docker images. Since nix
itself of course runs fine inside docker, I suspect you should be able to
build a docker image on Windows that way, too, but I haven't tried.

Also, even if you come up with a forward facing solution and demonstrate it
only on a particular domain you still, IMO, deserve most of the credit for
solving the problem even if it takes others to translate it straightforwardly
to other domains.

~~~
pjmlp
Nix only "works" in macOS, in the context of pretty UNIX, software development
on macOS is usually something else.

~~~
vthriller
I'm not sure I'm following. Isn't this argument applicable against Docker as
well?

~~~
pjmlp
No, Docker on Windows can make use of Windows containers, and also
interoperates with any kind of Windows or macOS application.

~~~
patrec
In what way does docker on macOS interoperate better with the rest of the
system than a nix package? I'm pretty sure the answer is going to be "none" –
you can build native, including UI apps with nix (not that I recommend
throwing away xcode for your app store development and switching to nix). How
do you do that with docker?

I'm less sure about windows, can you explain a bit more how you use docker
containers for providing "native" windows stuff (as opposed to as a more
lightweight linux VM replacement when developing something that you really
want to deploy on linux)?

~~~
pjmlp
For example deploying IIS based applications, including some Windows services
and COM libraries.

~~~
patrec
Thanks, I wasn't aware of this.

~~~
patrec
Out of curiosity, why are you using docker containers on windows? For emulate
static linking or to provide network/process isolation (and does docker under
windows provide "proper" i.e. secure isolation?) or something else entirely?

~~~
pjmlp
Configure a couple of application servers from scratch and manually
configuring the services starts to get tiring.

Usually the traditional option would be to snapshot the VM right after the
first successful install.

However stuff like Windows Nano now makes it interesting to start playing with
containers on Windows.

------
dragonsh
If you like Nix than probably try GNU Guix [1] and if possible use Guix System
on server side. This is one of the most modern development in operating
systems.

In the modern security conscious world where most companies run their
workloads on virtual machines controlled by other companies, it's imperative
that applications are deployed on such predictable, secure and reproducible
operating systems.

Guix supports transactional upgrades and roll-backs, unprivileged package
management, and more. When used as a standalone distribution, Guix supports
declarative system configuration for transparent and reproducible operating
systems.

I still think Amazon, Google and Azure will take a decade or two to build and
offer something like this to it's customers. Indeed Google's Fuschia is trying
to do the same, but I feel it's at least a decade away.

It has one of the best wonderful documentation [2] to get started and explore
with all the details.

[1] [https://guix.gnu.org/](https://guix.gnu.org/)

[2]
[https://guix.gnu.org/manual/en/html_node/](https://guix.gnu.org/manual/en/html_node/)

~~~
sterlind
As a Nix user, Guix interests me but I have three concerns, namely my pain
points with Nix:

1\. Small package base.

My workflow includes running Julia on CUDA, editing from VS Code. Remarkably,
Guix has both Julia and (on a third-party PKG repo) CUDA. What it's missing is
VS Code. I understand that VS Code is non-free (like Chrome, it attaches
trademarks on its branding), but there's no VS Codium I could find either. I
also can't find Atom. Am I missing something?

2\. Package index mirroring.

Nix has an issue where Python, Node and other "third-party" packages have to
be manually added to nixpkgs. If you want celery or requests, you're in luck
because those are popular. If you want something more obscure, like pywikibot,
you have to add it yourself (or run pip/npm yourself, and forego the benefits
of Nix/Guix.) Does Guix address this?

3\. Ease of scripting.

This might be a plus for Guix, since Nix design patterns are arcane (the
language is easy, things like overrides and needing to replace broken URLs is
hard), but how easy is it to debug failing builds? Is there something like a
debugger, or at least a way to shell-in and call things yourself?

Bonus #4: broken URLs. is there a cache? what can be done?

~~~
rkangel
The solution to Python packages is the 'pypi2nix' tool. This is an alternate
approach that you see also used in the blog post, this time with vgo2nix.
These tools take a set of dependencies (e.g. requirements.txt) and generate
Nix expressions to build all of them from source.

~~~
adisbladis
You may also want to take a look at [https://github.com/nix-
community/poetry2nix](https://github.com/nix-community/poetry2nix)

Disclaimer: I am the author (and also the author of vgo2nix).

------
Legogris
I've been using NixOS on my laptop for a couple of years now and I am still
embarrasingly undereducated on how to do some of even basic things. I feel I
still haven't fully grokked "the nix way". You will need to understand it
pretty well to do even basic things like running specific versions of ruby or
python with specific native libraries. I have spent a couple of hours here and
there on nix pills but I think one really needs to set aside some concerted
time to get into it properly. I consider myself fairly proficient with Linux
in general and never had this kind of steep learning curve with other distros.

(This is a not me criticizing NixOS, just noting that they do things very
differently and you really need to invest some time and effort to start being
able to solve your own problems rather than relying on what's provided by
others)

~~~
newnewpdro
Can't you just throw it in a container to explore and learn the ropes before
pivoting your entire host?

~~~
takeda
Not sure I understand your question, but NixOS does have option to test things
"nixos-rebuild test" (to test without saving it permanently, so reboot should
restore it) or "nixos-rebuild build-vm" (which builds a vm image with the
config)

~~~
newnewpdro
[https://wiki.archlinux.org/index.php/Systemd-
nspawn#Examples](https://wiki.archlinux.org/index.php/Systemd-nspawn#Examples)

I don't see a Nix example there, but the NixOS github looks to have some
issues already surrounding provisioning Nix for use in containers.

Containers are a great way to explore a distro without messing with your host,
you can at least check out their user space, filesystem layout, and package
tooling. The main shortcoming is you don't get to see any kernel-specific
features, since you're essentially chrooting into the other distro's rootfs.

There's always VMs too...

~~~
takeda
I haven't used NixOS outside of VirtualBox, but plan to use it for my next
Linux setup.

I believe the reason for build-vm is that systemd is central part of NixOS and
you can't test it by starting it inside of a container (I don't think systemd
even allows it).

The nixos-rebuild command actually creates a path with a new configuration, so
you could similarly use nspawn and go there. When you use "nixos-rebuild
switch" it updates "/run/current-system" symlink to that new location.

~~~
newnewpdro
systemd is container-aware, you should be able to directly "boot" nix
containers then using tooling like systemd-nspawn, and login/manage using
machinectl. It will all "just work" with systemd on both sides.

~~~
takeda
I see, was not aware of it, although perhaps it might be possible to do it
now:
[https://github.com/NixOS/nixpkgs/pull/67232](https://github.com/NixOS/nixpkgs/pull/67232)

------
jolmg
It's been years since I last tried NixOS. One of the things that I really
didn't like was that changing something that had a lot of dependencies meant
that everything that depended on it, needed to be rebuilt. That's so even if I
knew the change couldn't affect how the dependencies would be built.

I know why NixOS requires that. I think that's really cool from a safety side,
but it was also really annoying to have to wait so, so long to use the system
after an insignificant change. I wish there were a "I know what I'm doing; let
me risk screwing up" mechanism. I don't know if there's one now, but I kind of
doubt it.

I imagine Guix would have the same problem.

~~~
Ericson2314
You want
[https://github.com/NixOS/rfcs/pull/62](https://github.com/NixOS/rfcs/pull/62).
I'm very eager to see it done this year (ideally earlier but don't want to
jinx myself).

~~~
bb010g
Holy shit, thanks for linking this.

~~~
Ericson2314
Sure! I'm very excited too :)

------
oldwinebottles
Happy to see more people interest in Nix. At the end of the post there are
some links on learning resources, that list is 100% the best way to get
started.

~~~
thundergolfer
I'd also highly recommend reading at least the first two parts of the Nix
author's PhD thesis: [https://grosskurth.ca/bib/2006/dolstra-
thesis.pdf](https://grosskurth.ca/bib/2006/dolstra-thesis.pdf)

It really clearly explores the problem space Nix dives into. I learnt a lot
about why Nix needs to exist.

------
zegl
Setting CGO_ENABLED=0 does not make the build fully static. To do this you
currently need to set the following flags:

    
    
        -ldflags '-extldflags "-fno-PIC -static"' -buildmode pie -tags 'osusergo netgo static_build'
    

See
[https://github.com/golang/go/issues/26492](https://github.com/golang/go/issues/26492)

------
anonsivalley652
If they built their app differently, such as using a temporary directory
rather than building it in tree in the release path, they could've used staged
builds to cache their steps in docker. What they do get from Nix package
management is freedom from dependency hell.

Actually, I prefer habitat (hab) for portable Linux packaging because it
supports a crazy amount of caching, it does GC on unneeded packages, it has an
internet-visible package repo to publish to and has a simple DSL that I can
teach to anyone. I don't use its other features that much, and I don't use it
on FreeBSD but I use hab studio on macOS now and then.

Nix is pretty awesome because it _almost_ marries package management and
configuration management... if you think about it, the definition of a
conventional OS package is an arbitrary one that limits configurability and
customization. Gentoo, Arch and others try to work around it, but you really
can't unless you can drive and hook intelligently into specification language
that built the package and can accept modifications at different points
without a lot of kludgy hacks or forks. Furthermore, the conventional package
paradigm assumes one version and one configuration for everything, unless you
want to get messy with variant packages. Nix and hab solve these issues.

The problem is I can't suggest it in a lot of usage contexts because it would
be like if I recommended Rust, Elixir or Haskell to a PHP shop. ;) I really
want to but I know it's probably not a good idea. Nix is awesome wherever the
badassometer says "above average." Give it a try, but use what works best.

Best practices > Leverage build cache:

[https://docs.docker.com/develop/develop-
images/dockerfile_be...](https://docs.docker.com/develop/develop-
images/dockerfile_best-practices/#leverage-build-cache)

Multi-stage build:

[https://docs.docker.com/develop/develop-images/multistage-
bu...](https://docs.docker.com/develop/develop-images/multistage-build/)

~~~
ses1984
>If they built their app differently, such as using a temporary directory
rather than building it in tree in the release path, they could've used staged
builds to cache their steps in docker

Can you explain this just a bit more, please?

I was just reading the two articles you linked today, and your comment seems
there is some piece of insight that's missing.

If you search the multi-stage build article for "cache" the only hits are
related to _not_ using cache.

What should I do if I want to leverage the cache as much as possible in a
multi stage build?

------
candiddevmike
I've tried to use Nix in the past, but I can't get past the layers of
abstraction. Normally I go and consult the relevant docs for what I'm trying
to configure (postgres, systemd, etc), but with Nix I also need to consult the
Nix docs and figure out the discrepancies/map values between them. I love the
idea of systems configuration like this, but I'd much rather see less
abstraction ala cloud-init with nixos-rebuild.

For those who use Nix, how do you deal with this kind of abstraction?
Additionally, how soon do you see security updates land?

------
srik
Caution for OS X users —
[https://github.com/NixOS/nix/issues/2925](https://github.com/NixOS/nix/issues/2925)

~~~
anarchogeek
I find it pretty stunning that Nix has decided to drop support for OS X over
ideological reasons like this. There are workarounds, but the reality is that
they're upset at apple's decision so they just dropped support. We're not
close to a year in to this these issue threads. Any software which would be so
anti-user does not deserve to be used. Nix might be magical but that sucks.

~~~
user555555555
Can you provide a link to where it says they have actually droped support.

In that link they describe how to make a firmlink, its _seems_ like they have
a way to just keep doing their thing as before

(edit, reworded)

~~~
whateveracct
It's FUD

------
kragen
Yeah, Nix (or Guix) is kind of Docker Done Right.

~~~
Sean1708
I think it's a lot more complicated than that. Yes there are plenty of people
that use Docker as a package manager, but I think that's a failing of
traditional package managers rather than a failing of Docker. You could, for
exactly the same reasons, say that Nix is Nix Containers[1] Done Right but if
that were true then Nix Containers wouldn't need to exist in the first place.

[1]: [https://nixos.org/nixos/manual/#ch-
containers](https://nixos.org/nixos/manual/#ch-containers)

------
h91wka
I have to do a bit of DevOps at work, and I absolutely hate every second of
it, hence I tried Nix, and I think it is absolutely awesome. As a software
developer, I find the ability to take a cryptographically consistent closure
of a program's environment and deploy it in a reproducible way across your
dev, loadtest and prod machines _the only reasonable way_ to do things. Not to
mention that Nix community is very knowledgeable and open to patches and
suggestions.

However after a brief investigation period I abandoned this idea altogether,
here's why.

In my opinion, Nix is only worth climbing its steep learning curve when you
really utilize its "reproducible environment" concept. As a counterexample,
take your home PC environment, which is 1) unique 2) quickly changing. You
don't need to reproduce it very often (how often do you upgrade your laptop?).
You don't really want a hash-perfect reproducibility there, and a simple
Ansible script will work just fine. So, the target of Nix should be "serious
enterprise", where you deploy hundreds of machines, right? Well, "serious
enterprise" where I am at (banking) will _never_ adopt Nix. There are two main
reasons why:

1) NixOS doesn't follow standard Linux filesystem hierarchy. They get around
this by patching everything, both manually and automatically. nixpkgs repo
contains thousands of patches. To put it in the perspective, OS is one of your
trust anchors. I don't want to sound as "no one got fired for choosing IBM"
guy, but legal people will scream bloody murder if they see extensive level of
patching going on in the distro, for a variety of reasons, not necessarily
tied to security. Some of their patching goes very deeply, BTW, and straight
up changes behavior of the software.

2) Secrets management: it doesn't exist in Nix. nix store itself is world-
readable. There are fundamental problems with some cryptographic algorithms
that require entropy, and therefore go against "reproducible environment"
concept.

YMMV, but for me Nix didn't do the job, although I studied it deeply, and
contributed some patches to nixpkgs. It looks like a futuristic research
project. On the outside it is brilliant, but in brutal reality you need to run
the ugliest scripts to patch shebangs and what not.

~~~
fctorial
> Some of their patching goes very deeply, BTW, and straight up changes
> behavior of the software.

Can you give some examples?

~~~
h91wka
Some language-specific build systems get their ability to work with the online
repositories deliberately broken.

------
tlrobinson
> The image age might look weird at first, but it’s part of the
> reproducibility Nix offers. The date an image was built is something that
> can change with time and is actually a part of the resulting file. This
> means that an image built one second after another has a different
> cryptographic hash. It helpfully pins all images to Unix timestamp 0, which
> just happens to be about 50 years ago.

This doesn’t mean builds done by Nix are bit-for-bit reproducible by default,
does it?

There are a lot of other ways to introduce non-determinism in builds, like
rand() or network requests (which I think could only be eliminated in a
generic way by literally emulating the CPU of the machine doing the build and
not allowing external communication?)

~~~
fctorial
It provides the hash of files as a method of verifying the input identity. As
long as the build file is pure function of build tools (gcc, make, etc), and
the build tools themselves are deterministic, the build outputs should be
reproducible. Though I've never seen anything like `-frandom-seed=<input-file-
name>` in build files. I think bit-by-bit reproduciblity is one problem
tackled by guix.

~~~
danieldk
_I think bit-by-bit reproduciblity is one problem tackled by guix._

I think the Nix folks are also interested to some extend. Currently, almost
99% of the paths in the minimum installation image are reproducible:

[https://r13y.com/](https://r13y.com/)

------
eximius
I still haven't figured out the nix language, how to find which package
installs a particular binary if it isn't the name of the package, or how to
find config options for packages I do have.

Those three problems are what's holding it back for me. I still enjoy it so
far, though.

~~~
SitiSchu
To find out which config options exist for packages searching here can be
quite helpful:
[https://nixos.org/nixos/options.html](https://nixos.org/nixos/options.html)

Another way would be reading the nix expression on the nixpkgs GitHub
repository.

------
whateveracct
I use Nix for every project once it grows enough for me to have to pin a
directory. And sometimes sooner.

I don't know it that well but I've picked up on design patterns, use the repl
to figure stuff out, write small abstractions when necessary..and it works
pretty great. And the gains grow as my dependencies become polyglot (even C +
another language like Haskell..but when you add compile-to-JS, static asset
generation into the mix it's amazing)

Yeah it's hard to learn. No denying it. But every other "alternative" I've
found doesn't actually approach it. So I've kind of gotten sick of reading
about "alternatives" because it always feels like they don't actually solve
the problem Nix solves. I always end up using the alternative+Nix. Classic
example of this is the Haskell stack build tool.

I could've bemoaned complexity and various capitalist criticisms when I first
ran into it at a job. But I'm so much better off for not.

------
sagebird
I am not familiar with the topic at hand.

I know that include os stuff is young.

But I wonder, would it be easier to build the website with go, with an os
included, and then deploy that?

Both the docker and nix approach seem like alot of random tasks or knowledge
is required. Are there bad things about including an OS I am forgetting?

------
kalium_xyz
The effort involved in using nix becomes a lot less if you make templates and
have IDE support.

------
daenz
Most of their complaints about Docker, specifically:

    
    
      * The package manager is included in the image
      * The package manager’s database is included in the image
      * An entire copy of the C library is included in the image (even though the binary was statically linked to specifically avoid this)
    

Can be solved with Docker multi-stage builds[0]. This is essentially a
separate build stage inside your single Dockerfile that sets up, builds, and
creates artifacts. Those artifacts are then copied over into a later build
stage. The resulting docker image is lean and yet the artifacts were built at
docker build time, and its all contained in a single file.

0\. [https://docs.docker.com/develop/develop-images/multistage-
bu...](https://docs.docker.com/develop/develop-images/multistage-build/)

~~~
takeda
The author did use a multi stage build.

~~~
xena
A ridiculously slim multi-stage build, but it was starting to show its
weaknesses.

------
dfc
> This article was posted on 2020 M2 10.

What is "2020 M2 10"?

~~~
_Nat_
Looks like that author prefixes the month with " _M_ ". Dunno why.

Sometimes DD-MM-YYYY and MM-DD-YYYY can be confused, such that if one of those
formats is used, it could be helpful to clarify. However, YYYY-MM-DD is
typically pretty unambiguous, so it seems odd here.

I guess it's probably meant in analogy to, e.g., " _2020-Q1_ " (which would be
the first _Q_ uarter of 2020) or " _2020-H1_ " (which would be the first _H_
alf of 2020).

~~~
xena
I do it because it's visually unambiguous. It is clear what is the day number,
what is the month number and what is the year number. I have no idea where
Apple got this, but it's the date format used in the iOS Lojban locale. I
kinda got used to it because it's also faster for me to type.

------
pmarreck
It is honestly great to have a tool that solves so many problems at once but
the resultant complexity is intimidating for most developers I feel.

~~~
takeda
This is true, it takes quite a bit of effort to learn it. I think what Nix
needs is perhaps series of tutorials targeting how to solve common problems
that nix is useful for. There are plenty of things that are abstracted in
nixpkgs, so perhaps use them as building blocks and explain how they are
implemented in different articles.

------
aliswe
> And the output is 16 megabytes smaller.

Am I missing something here? Is this the point? 16MB of saved space?

------
kstenerud
My only foray into nix was cut short because my 2gb machine RAN OUT OF MEMORY
installing a package.

~~~
fctorial
Well, it's not just a wrapper around "tar xf"

------
krick
What's even the point of top-level domain "website"?

------
axilmar
After reading the web page posted above, and the comments in here, I've
reached the conclusion that Docker, Nix, etc are trying to solve a problem
that shouldn't exist, and that the actual problem is at the operating systems'
provided mechanisms for dealing with programs and dependencies.

More specifically, it seems like the Unix filesystem layout, with its bin,
etc, usr and other well-known directories is part of the problem.

Also, the environment variables mechanism is part of the problem.

Why couldn't we have a much simpler mechanism for running things? for example:

a) each application should live in its own directory.

b) each file or folder should be versioned.

c) common files for applications, for example, fonts, should be installed in
the application folder themselves, and the filesystem shall provide hooks for
external systems to be notified when these files are created, moved or
deleted. With this system, important files, for example tool executables,
would automatically be found and invoked without the need for adding a path to
them.

d) runtime program dependencies should be provided via text file with
parameters and not by environment variables. This text file would have to be
specified each time a program runs, and the user could create specific scripts
that run the program with specific parameters. Programs should have attached,
in the same folder, the default parameters file. How's that different from
environment variables? well, with this system there is no system-wide
dependency of a program to a unique string, and therefore the requirement to
override environment variables for specific programs wouldn't exist. I.e.
programs shouldn't need environments to run, but list of dependencies.

With the above solution, making repeatable and sharable development and test
environments would be extremely easy: just copy the application folders and
the configuration files one needs. If something is missing, copying it from
another source would be the solution.

Ultimately, having a set of programs at hand and wanting to combine them to
produce some output is exactly the same as having a bunch of functions in a
program and invoking them in a specific order to get some result. And while in
programming we have recognized the problem of global state and we have taken
measurements to minimize the problem, for example with functional programming,
we haven't done so for programs.

In other words, the problem is that we are trying to compute things using
global state, once more, this time at the operating system level!

For me, that's the fundamental issue. All these tools are welcomed, but unless
the fundamental problem is solved, no real progress shall be made and there is
always going to be tool fragmentation (i.e. do we use Nix? Docker? Some other
solution? etc).

~~~
leonnn
> the actual problem is at the operating systems' provided mechanisms for
> dealing with programs and dependencies. More specifically, it seems like the
> Unix filesystem layout, with its bin, etc, usr and other well-known
> directories is part of the problem. > > Also, the environment variables
> mechanism is part of the problem

The way I see it, Nix exactly solves the "fundamental problem" as you describe
it!

------
hardwaresofton
I still don't see the light. I'll try and lay out my reservations, starting
with the pain points of the docker image (out of order to coalesce some
points):

> The package manager’s database is included in the image > Most of the files
> in the docker image are unrelated to my website’s functionality and are
> involved with the normal functioning of Linux systems

So remove it...? Docker multi-stage builds[0] make this really easy, though I
still use separate containers for different times (building, testing, etc),
and sometimes even multiple "stages" of containers for languages that take a
long time to build (Haskell).

> The package manager is included in the image

Assuming you don't want the package manager itself, why not just use a
scratch[1] or distroless[2] image?

Well of course it's harder than one might expect to use those options
(distroless is actually fine), because of the lie of "statically building"
anything on most distros other than alpine, which brings us to the next pain
point...

> An entire copy of the C library is included in the image (even though the
> binary was statically linked to specifically avoid this)

Using alpine gets you much closer (if not all the way there) to a proper
static binary w/ musl libc. You may have to look into things like replacing
the use of libnss or certain systemcalls with golang libraries, however[3].

My biggest problem with nix is that it just _isn 't worth_ the effort -- The
~5+ paragraphs of nix tools and scripts is just not at all attractive to me
personally when I know roughly a day's worth of fiddling to get a proper
static build is good enough most of the time. I've heard amazing things about
nix, nixops, and all the tools there-in but there just isn't enough pain IMO
to warrant completely changing how to do things. Nevermind the fact that disks
get cheaper and cheaper, network gets faster and faster, and people are
starting to look into pre-emptively sharing container images across meshes of
nodes with tools like Dragonfly[4].

Another point that is somewhat related -- if nix is good, guix has to be
better on some level (excluding factors like ecosystem) purely because it uses
a full-blown language for the config (i.e. terraform vs. pulumi).

It doesn't seem like nix will hold enough value before unikernels "arrive",
then the problem of taking your distribution along with the program you want
to deploy disappears alltogether.

[0]: [https://docs.docker.com/develop/develop-images/multistage-
bu...](https://docs.docker.com/develop/develop-images/multistage-build/)

[1]:
[https://hub.docker.com/_/scratch/?tab=description](https://hub.docker.com/_/scratch/?tab=description)

[2]:
[https://github.com/GoogleContainerTools/distroless](https://github.com/GoogleContainerTools/distroless)

[3]:
[https://github.com/golang/go/commit/62f0127d8104d8266d9a3fb5...](https://github.com/golang/go/commit/62f0127d8104d8266d9a3fb5a87e2f09ec8b6f5b)

[4]: [https://d7y.io/en-
us/docs/userguide/download_files.html](https://d7y.io/en-
us/docs/userguide/download_files.html)

~~~
Nullabillity
> It doesn't seem like nix will hold enough value before unikernels "arrive",
> then the problem of taking your distribution along with the program you want
> to deploy disappears alltogether.

Unikernels still don't solve the build problem, only the distribution. And you
can get that workflow today, by shipping around container or VM images. And
Nix can even build either for you!

~~~
hardwaresofton
Correct me if I'm wrong, but they do solve the build problem -- you can build
static libraries way easier if you replace all the system stuff libc was doing
for you with approaches like unikraft[0] or rumprun[1].

[0]:
[https://xenproject.org/developers/teams/unikraft](https://xenproject.org/developers/teams/unikraft)

[1]: [https://github.com/rumpkernel/rumprun-
packages](https://github.com/rumpkernel/rumprun-packages)

~~~
Nullabillity
I don't have time to look at rumprun right now, but from the Unikraft page:

> The Unikraft build tool is in charge of compiling the application and the
> selected libraries together to create a binary for a specific platform and
> architecture (e.g., Xen on x86_64). The tool is currently inspired by
> Linux’s kconfig system and consists of a set of Makefiles. It allows users
> to select libraries, to configure them, and to warn users when library
> dependencies are not met. In addition, the tool can also simultaneously
> generate binaries for multiple platforms.

So it doesn't solve getting the makefiles (or presumably the source code
depending on how it's organized), the compiler, or assembling multiple
projects into one coherent build.

Using Make as the big unifier also sounds a bit scary, since it's so easy to
screw up dependency lists or introduce accidental impurities, because it has
no way to verify either.

~~~
hardwaresofton
> So it doesn't solve getting the makefiles (or presumably the source code
> depending on how it's organized), the compiler, or assembling multiple
> projects into one coherent build.

Ahh I see what you mean -- I was more thinking about the library issue, but
yeah building is definitely still really hard.

I think good support for unikernels is a long time off, and will obviously
vary by language -- maybe the usage that finally breaks through will be tight
integration with some lower level compile tools. For example if a company were
to hit GraalVM (which is really trying to push themselves as the better-than-
stock VM for a bunch of languages) and LLVM as integration points for
unikernels I think they could really make a convincingly easy toolchain
(without modifying developers' current toolchains).

------
prpl
A little disingenuous to compare without docker multi-stage.

~~~
infinisil
They are using a Docker multi-stage build in the very Dockerfile they showed..

~~~
mappu
Yes - but then their final stage is built on top of an Alpine container, and
they complain about Alpine's package manager + OS files being included in the
image.

If their final stage was based on scratch or distroless, the Docker file size
would have come out to the minimal ~90MB result too.

~~~
_-___________-_
Yeah, I don't understand that part either. If you want to be able to shell
into the container and poke around, use Alpine by all means, but then don't
complain about a C runtime being included. If you want a minimal container for
your static executable, use scratch.

------
dr01d
I would recommend multistage build but with distroless as final.
[https://github.com/GoogleContainerTools/distroless](https://github.com/GoogleContainerTools/distroless)

~~~
fcarraldo
This seems like the real answer to this problem. I really like Nix
conceptually, but the problem it's being used to solve in this case is already
solved by Docker, and using a multistage build with a Distroless stage relies
on fewer dependencies, fewer tools, and many fewer lines of config.

------
Annatar
This is so wrong on so many levels. So many complications. Monstrously huge
binaries. Static linking. Docker and Nix mess of complex instructions, all
custom. With RPM, it is one command, rpmbuild, with a standardized spec file
and no need to link statically. No Docker image needed, or anything else. Just
plug into a yum repository or into a Kickstart install group and you're done.
No code or compose files or playbooks to write. Turn the VM on and have it
automatically PXE / DHCP / TFTP boot and Kickstart install it with the desired
software and configuration. Without writing a single line of code or lifting a
finger.

~~~
Nullabillity
And now your setup depends on working PXE (which requires custom DHCP config!)
and a custom RPM registry. How is that less complex operationally?

And you need to write custom rpmspecs and kickstart manifests. Do those not
count as code?

Kickstart also doesn't help you with any day-two ops concerns.

~~~
Annatar
"Custom DHCP config"?!? You do realize DHCP stands for "dynamic host
configuration protocol"? That's exactly what it was designed for... And if you
can't get "working PXE" (PXE is implemented in firmware, by the way), you
might want to _brush up on the basics_ of putting together an IT
infrastructure, or this moment might be a good time for a career switch out of
IT... There are no "Kickstart manifests", it's a ruleset. There is no "RPM
registry" (this isn't Windows™️) or anything of the sort. There are no "custom
spec files" as it's a formalized format with grammar and lexicography which
the OS's software management subsystem understands - it's like putting a VHS
tape into a video cassette recorder, modular and standardized.

Day two is where a software deployment server and clearly defined change
management process according to the department of defense capability maturity
model come into play.

~~~
nrr
The PXE client is implemented in firmware, but for the PXE client to have
anything to talk to, there's a sizable amount of moving parts to configure
just right in order for it to work well.

Those moving parts often include telling your dhcpd to include some additional
payloads to offer to clients when they obtain IP leases. This is in tandem
with getting tftpd running, writing your ks.cfg, and writing a suitable
pxelinux.cfg so that you can boot into Anaconda to install the OS.

This is all assuming that you even have the appropriate access to do these
things in our increasingly cloud-centric world. Amazon, Microsoft, and Google
all won't let you use PXE to provision VMs. Notwithstanding, networks that are
IPv6-only pose a problem for netboot installations. PXE doesn't work so well
there unless you make the conscious decision to use DHCPv6 over stateless
autoconfiguration.

\--

Now, this all said, you're on something resembling the right track with
respect to OS packages. There's a common pattern (at least in my experience)
in the public cloud world where you use something like Packer to build a
machine image for provisioning a VM in, e.g., EC2. I've used Kickstart to
great effect here in conjunction with some rules in the build system for
autogenerating specfiles to hand off to rpmbuild.

Combine that with a set of functional and integration tests (the apps I wrote
during this period of my life were in Ruby, so the tests were in Ruby, but the
technology doesn't matter; you could just as well use ksh93 or pdksh) to
ensure that the image you built is what you expect, and it's actually rather
killer.

The only things that yum/rpmbuild truly lacks that Nix gives us for free are
(1) a package build environment, on Linux at least, that's hermetic all the
way down to libc; (2) the ability to install more than one release of a
particular library at a time without confusing the dynamic linker machinery;
and (3) a way to herald when to rebuild certain packages based on changes to
their dependencies.

Apropos deployment and running in production, the intent then is to keep it
immutable (you disallow SSH as a general rule after day one, right?) and for
your change management process to consider the VM as a whole deployment unit.
Want to deploy updated configuration? Deploy new VMs. Want to deploy a new
software release? Deploy new VMs.

~~~
Annatar
SSH is not disallowed, because there are still cases where it is necessary to
ssh in order to service faulty hardware. However, ssh is disallowed for making
any ad hoc changes to any system, unless that is done by removing, installing,
or upgrading a configuration OS package. It would be better to say that any
commands which manipulate system state by hand are disallowed. In worst case
scenario, every system is a throwaway one: since the configuration is 100% OS
packaged, I can just re-instantiate a server, be it physical or VM. I don't
have AWS or hosting trouble because I didn't fall for external hosting hype,
and having everything system engineered, I run everything on-premise for
peanuts. AWS or Azure just cannot compete with me in cost.

~~~
nrr
I'm left with just questions at this point. Assume production, not development
or UAT or staging or integration, etc.

How do you enforce that kind of compliance on SSH sessions? How do you audit
for SSH sessions that invoke commands that mutate the state of the system? How
do you account for configuration drift in the event that an SSH session
mutates machine state outside of those compliance and auditing mechanisms?

Why are you SSHing in to curate installed packages on machines manually
instead of letting a deployment agent/service take care of that?

Furthermore, what kind of environment are you operating in where your response
to a hardware fault is to SSH in instead of immediately evacuating the
workloads from the faulty machine, replacing it with a new one, and taking the
machine to a triage location for further investigation?

Do you operate at a small enough scale where leaving a faulty machine active
on the network, installing packages by hand, and SSHing to individual machines
is actually sensible?

Do you have a lot of false hardware alarms where the response is to SSH in,
run a few commands, and then bail when the going looks good? What kind of
monitoring practices do you employ?

~~~
Annatar
SSH in and get caught messing with the system manually - get fired. Keeps
everyone honest.

The environment is close to 90,000 systems. We service the hardware because it
is configured redundantly, for example the / filesystem is always on a mirror.
The physical systems are system engineered and configured to withstand
multiple hardware faults without any loss of service.

~~~
nrr
You keep saying these things, but I'm less and less convinced that you have a
significant hand in how these things are "system engineered," as you put it.
I'm also concerned by how few of my questions you actually answered.

"SSH in and get caught messing with the system manually" is an extremely hand-
wavey answer, especially in an environment of O(1e6) machines. I'd expect such
an environment to have a rather significant degree of automated compliance and
audit controls in place.

You'll also note well that I didn't say not to service machines. I'd asked why
you would prefer to leave a machine with an implied potential data-destroying
fault in the rack than immediately swap it out with a new machine that has
been verified not to destroy data. The servicing part comes into play here in
order to mark the previous-faulty machine as no longer faulty.

In particular, rack space is expensive, and certifying a machine as fit for
service again can take a long time, so it's a bit of a waste of real estate to
leave a machine that can't serve production traffic in place when you can
quickly swap it out with one that can.

Furthermore, redundancy doesn't help when you have an HBA or an ethernet
controller or a memory controller or CPU or a piece of PCIe interconnect
hardware or a hard disk with a firmware bug that corrupts data. At that point,
your redundancy becomes a liability, and you could wind up propagating corrupt
data throughout your system.

This all said, I'll agree that the louder and more obvious hardware faults
like disk crashes are relatively easy to cope with, so in those cases, I'd
likely also leave the machine in the rack and just perform the respective
component swaps. The place where I predict we'll disagree is whether to
evacuate the machine's workloads prior to servicing.

So, again, I'll assert that you likely have less of a hand in the engineering
of these things than you're letting on. That's nothing to be ashamed of, but
it does make your responses seem less genuine when you try to pass off your
experience working in such an environment as expertise in designing such an
environment.

~~~
Annatar
I have single handedly designed entire data centers, irrespective of which
impression you might get from my responses, which are admittedly terse because
I'm usually on a mobile phone tap-tapping them out, and that starts to
severely piss me off really quickly. Like now. Which is why I'm going to stop
right here.

