
NixOS and Stateless Deployment - idlewan
http://gfxmonk.net/2015/01/03/nixos-and-stateless-deployment.html
======
krick
I tried NixOS once and the overall impression was: it's magic. I mean, you
know, when you are installing tools on relatively "clean" distro, like Arch,
there usually something fails, and you have to try this and that, tweak
something, and it is good old "linux way".

And NixOS is something so elegant and beautiful, so mathematically "right",
that I subconsciously expect that it wouldn't actually work, and yet it does.
I tell it what I want to get (relatively marginal configuration, btw) and it
configures installation just right. I break something and it rolls back
without a problem.

So the main question I've had ever since: why this stuff isn't popular enough
yet.

~~~
nextos
Nix is nice. However, some practical aspects are (still?) quite ugly. It lacks
decoupling between packages and optional runtime dependencies. You can disable
optional dependencies, but this would lead to a different package hash
negating the use of prebuilt binaries.

Therefore, the culture seems to have all default package builds with all
optional dependencies on. This leads to situations such as installing mutt and
getting python too! (mutt -> gpgme -> glib -> python)

Last time I checked, if you installed git, you'd also get subversion, etc.
Quite sad, given that nix is full of so many fantastic ideas. Hope it matures
soon.

~~~
ArchD
I think you could define your own mutt package that doesn't have the optional
stuff, although it may be a lot of work.

~~~
barkmadley
I don't know much about the nix language, but it looks to me like the gpgme
support is optional:

[https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...](https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/mailreaders/mutt/default.nix)

~~~
nextos
The thing is easy to fix. The problem is that (i) most packages are far from
minimal (ii) you loose pre-built binaries.

~~~
barkmadley
True enough. I wonder how much of the problem comes from trying to be
compatible with software packages designed for other systems.

------
jdreaver
I'm a mechanical/software engineer, but I set up our company's web and
continuous integration servers using Ansible. The experience overall was
excellent, and everything worked mostly as advertised. A couple pain points I
had were:

* If I removed something from the Ansible configuration, it stayed on the server unless I explicitly removed it manually. This created hidden dependencies. I solved this problem by creating a brand new server and running Ansible on it from scratch every so often. I have considered setting up CI for our Ansible configs by using Vagrant to recreate our server architecture, running Ansible on the virtual machines, and ensuring everything works.

* Our continuous integration setup requires Ansible to be installed on the CI server, so it can automatically deploy to staging using the same playbook (Ansible configuration) we use for deployment. Our staging server is the same as our CI server, and it was actually a pain to set up deploying locally as root. Also, I feel like allowing the CI software to use root is a security hole.

I also spent some time with NixOS a year ago, and I was very impressed with
how it manages packages. The first problem I mentioned with Ansible seems like
it wouldn't happen with NixOS, since not including a package in an environment
means it won't be present. Second, it also seems that you could use Nix's
declarative configuration language in restricted environments, which wouldn't
necessarily require root, instead of having to install system-wide packages
for a particular deployment. I am not sure how easy this is in practice.

Currently, I am using Arch Linux, and I installed the nix package manager to
play with some more. In the future, we might be provisioning AWS servers in
real-time to run simulations given to us by customers (we make simulation
software), and in that case I am going to investigate NixOS more.

~~~
falcolas
Most orchestration tools are stateless from run to run, and as such can't
identify if a package has been removed.

The solution in my experience is to change the state to absent instead of
removing it from the playbook, and then refactor it out at some point in the
future.

~~~
mercurial
> Most orchestration tools are stateless from run to run, and as such can't
> identify if a package has been removed.

But NixOS is stateless from update to update as well. The key difference lies
in the difference of scope between what is described in a playbook and what is
described in a NixOS configuration file.

~~~
falcolas
I'm not talking about nix here, I'm talking about how to mitigate the problem
he encountered using Ansible.

------
gregwebs
This kind of discussion seems irrelevant once you move to the immutable
deployment model. A new deployment means a new server that the load balancer
switches to. Discard the old server. Obviously this doesn't work as well if
you are not on the cloud or otherwise not operating in a cloud-friendly (don't
write to the filesystem unless it is something like ceph) way. But I have no
idea why people are still updating cloud servers instead of replacing them.

~~~
cwp
Yes and no. It's true that immutable servers work around the flaws in puppet
et al.

However, nix still still really useful even when you have immutable servers.
Once the components of an app are described as nix packages you have a lot of
flexibility in working with them:

    
    
      • deploy to a local sandbox for development
      • deploy to virtualbox instances for end-to-end testing 
      • deploy to cloud instances for testing or production
    

..all from the same package descriptions. I've found it really valuable to
have a precisely-defined and exactly reproducible environment for my code,
whether in development, testing or production.

~~~
nXqd
deploy to sandbox, virtualbox o cloud instance is just a small change of ip
address in Ansible. This is convenient, but in my point of view it's not
really important. The way nixOS provisions new system is great, there is no
doubt about that. But the process of making an immutable system to me, it's a
bit tricky. In nixOS, for now you cannot debug a provisioned system without
doing it "properly". And it takes time. It would be great if nixOS has debug
mode for nix package manager, I think it's not so hard. For Ansible (and etc
..), you can debug and note it down, then change the playbook, destroy the old
one and up new one. It may cost the same time.

------
justinmayer
I have yet to find a single VPS provider that offers explicit support for
NixOS [1]. As much as I like the idea of NixOS, it's difficult to evaluate its
utility when I cannot easily spin up a NixOS-powered VPS and kick the tires.
There does not seem to be any support for NixOS on DigitalOcean [2], and the
instructions I came across for Linode look daunting enough that I am unlikely
to even try [3].

[1]:
[https://nixos.org/wiki/Hosting_providers_%26_NixOs](https://nixos.org/wiki/Hosting_providers_%26_NixOs)

[2]:
[http://digitalocean.uservoice.com/forums/136585-digitalocean...](http://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/4349028-support-
nixos-image)

[3]:
[https://nixos.org/wiki/Install_NixOS_on_Linode](https://nixos.org/wiki/Install_NixOS_on_Linode)

~~~
vertex-four
Pick any VPS provider offering KVM, and they'll likely offer the ability to
insert your own boot CD, from which you can install NixOS. I use DireVPS for a
small personal server, for example, although you'll have minor issues
installing (you'll need to configure the network from the command line before
starting to install, as well as specify the network configuration in the NixOS
configuration.nix separately).

You can also get NixOS running on EC2, if you happen to have an account:
[https://nixos.org/wiki/NixOS_on_Amazon_EC2](https://nixos.org/wiki/NixOS_on_Amazon_EC2)

~~~
mateuszf
Or any provider with Docker support.

~~~
listic
Do they explicitly advertise KVM or Docker support often?

~~~
vertex-four
If they're running one of those, yes. In general, if a VPS provider _doesn 't_
tell you what they're running somewhere (it's usually Xen or OpenVZ, KVM's
rarer, Docker's a new platform that doesn't really do the same things), you
want to stay away from them.

------
iElectric2
FYI: NixOS source is hosted on github:
[https://github.com/NixOS/nixpkgs](https://github.com/NixOS/nixpkgs)

------
otterley
Package management is great, and Nix is neat, but Nix isn't the entropy-
eliminating panacea the author makes it out to be.

Just like every other package manager, from rpm to dpkg, Nix is responsible
only for the aspects of the filesystem that it's specified to be responsible
for. It's not going to remove garbage left behind by users or poorly-written
post-install scripts, and it's not going to automatically undo other non-
filesystem-related state changes on package removals. And the practical
reality is that not everything on a system can be a native package anyway.

~~~
sparkie
Well, he does mention in the first few paragraphs that "it gets you out of the
notion of doing anything manually". The clue here is that Nix systems are
meant to be configured _only_ via Nix expressions. Any junk you throw onto the
filesystem manually is not part of the concerns of the rest of Nix, because
the software packaged by Nix can't even see that junk (when building software
or using a nix environment) - it can only see the dependencies specified in
the package definitions. The packaged definitions are what matters, because
those are the part you redeploy to other systems.

Any system is welcome to add it's own entropy, put it's own junk on - but when
you want reproducibility, you do it by specifying how to get that
reproducibility in your nix expressions.

~~~
otterley
One could argue the same for any package-oriented system, from Red Hat to
Solaris to Debian/Ubuntu ("only use packages"). The reality is quite
different, and it's always been different. I'm unaware of _any_ environment
that's used packages for every aspect of system management, and Nix isn't
going to change that.

Moreover, there's nothing stopping a Nix package author from writing a post-
install script that causes state changes to be made out of bounds. The package
manager isn't going to notice, and it's not going to clean it up at
deinstallation time.

~~~
sparkie
I think you misunderstand the architecture of Nix. Every package in Nix gets
put into an _immutable_ /nix/store. Packages are identified by a hash of their
contents, so a small mutation of any package definition gives it a new
identity, and thus, a separate package derivation.

Every package in the store has _exact_ dependencies - they reference the
hashes of other packages only. When a package is built with Nix, only the
directories for these packages are exposed to the chrooted environment in
which the build occurs - so it is simply not possible for some randomly added
junk in the filesystem to make it's way into a nix-defined package.

Contrast this to building software on another machine, where I might depend on
"glibc" version "1.0". The combination these two values hardly represents a
unique identity, as I could make any random package that fits those
requirements. It's much more difficult for me to create a package which
results in an hash collision though.

One thing that makes the other systems so unreliable is the presence of
multiple repositories. If you were going to deploy packages from a single
repository, then you can do careful planning in such a way that packages do
not have any collisions. Assuming no user mutates the directories under
control of the package manager, such system will also be _effectively
reproducible_. Current mainstream distros work surprisingly well because they
basically use this model, where a default repository provides most users
needs. These distros quickly break down when you start adding third-party
repositories which bundle alternative compilations of the same software that
sits in the "official" one.

Basing packages from hashes (identity), rather than names and numbers, and
making sure all of the files for each package is held in an isolated directory
ensures that collisions won't happen, even if two different repositories
provide the same software name and number, they are represented by different
hashes, and will be treated as distinct pieces of software.

~~~
retr0h
I like the architecture of Nix, and believe there are many benefits to it's
packaging model. However, I am not a huge fan of the syntax[1], and rather use
a template engine I am quite familiar with (like jinja).

I'll continue to follow and learn more about Nix, but anxiously awaiting my
epiphany.

[1]
[https://github.com/NixOS/nixpkgs/blob/1a504e3fb72fab10799cf0...](https://github.com/NixOS/nixpkgs/blob/1a504e3fb72fab10799cf07ba8eaa942bb7e585b/nixos/modules/services/web-
servers/apache-httpd/default.nix)

~~~
cwp
Yeah, problem with nix is that it's _different_. The nix language is a lazy-
functional DSL, which takes a bit of effort to learn.

The thing is, "better" does imply "different." The syntax is weird and
difficult, but it's the linchpin of the whole system. It's much, much better
than your typical packaging system and worth the effort to learn it.

------
falcolas
Please forgive me if I have trouble taking advice from someone who has set up
"tens" of servers and doesn't appear to understand the current set of
orchestration tools.

Don't get me wrong, Nix sounds great, but this is a poor article from an
inexperienced sysadmin who is unable to really point out the pros and cons.

~~~
mercurial
As a guy who is in no way a sysadmin but has devops experience, I thought he
nailed it pretty well. Take Ansible. It is declarative: you write down the
list of packages you expect to find on the target machine(s), and Ansible will
install any that is missing. Now, you remove one of these packages from your
playbook. Is it magically uninstalled from the targets? Absolutely not.
Because the playbook doesn't describe the "state of the world" in its
entirety. A NixOS file does that. Remove postgresql from a master NixOS
configuration file? Update and no more Postgres. Because everything that NixOS
manages is described (installed software, configuration, etc) (1).

1: Actually, last time I looked at NixOS, you could also install things the
imperative way, but that's a silly thing to do.

~~~
mSparks
I don't get it. Stick in the CentOS CD, click install, add the root user
account and password

And you're done. Server up and running. done about 20 of them in the last
twelve months. What's all this "package" stuff about? what's wrong with yum?

~~~
drinchev
Yeah that's what I was doing for almost 5 years before I found Ansible. Now I
even don't bother to put a CD. I go to my VPS provider and add a VPS then add
my key and run an ansible playbook, which installs whatever I want, the way I
wanted it.

The thing with package managers is that it get's boring to do everything all
over again for a large amount of servers. Also your brain memory does not have
a reliable version control. You need to keep why and how you do certain stuff
on your server somewhere else. First thing that comes to your mind is a shell
script and people are already using something better ( ansible, puppet, etc.
). Now, with this article, I see that there is something even better than
this.

