
Why Puppet, Chef, Ansible aren't good enough - iElectric2
https://www.domenkozar.com/2014/03/11/why-puppet-chef-ansible-arent-good-enough-and-we-can-do-better/
======
jes5199
A thing that this article is hinting at that I think might be more fundamental
to making good automation principles: idempotency.

Most of unix's standard set of tools (both the /bin programs and the standard
C libraries) are written to _make changes to state_ \- but automation tools
need to _assure that you reach a certain state_. Take "rm" as a trivial
example - when I say `rm foo.txt`, I want the file to be gone. What if the
file is already gone? Then it throws an error! You have to either wrap it in a
test, which means you introduce a race condition, or use "-f" which disables
other, more important, safeguards. An idempotent version of rm - `i_rm
foo.txt` or `no_file_called! foo.txt` would would include that race-condition-
avoiding logic internally, so you don't have to reinvent it, and bail only if
anything funny happened (permission errors, filesystem errors). I does _not_
invoke a solver to try to get around edge cases (e.g., it won't decide to
remount the filesystem writeable so that it change an immutable fs...)

Puppet attempts to create idempotent actions to use as primitives, but
unfortunately they're written in a weird dialect of ruby and tend to rely on a
bunch of Puppet internals in poor separation-of-concern ways (disclaimer: I
used to be a Puppet developer) and I think that Chef has analogous problems.

Ansible seems to be on the right track. It's still using Python scripts to
wrap the non-idempotent unix primitives - but at least it's clean, reusable
code.

Are package managers idempotent the way they're currently written? Yes,
basically. But they have a solver, which means that when you say "install
this" it might say "of course, to do that, I have to _uninstall_ a _bunch of
stuff_ " which is dangerous. So Kožar's proposal is somewhere in the right
direction - since it seems like you wouldn't have to ever (?) uninstall
things, but it's making some big changes to the unix filesystem to accomplish
it, and then it's not clear to me how you know which versions of what libs to
link to and stuff like that. There's probably smaller steps we could take
today, when automating systems. Is there a "don't do anything I didn't
explicitly tell you to!" flag for apt-get ?

~~~
sillysaurus3
Offtopic, but: what's an example of a situation where using rm -f is bad
compared to rm in practice? That is, an example where rm would save you but rm
-f would make your life upsetting?

On topic: idempotency may be a red herring in this context. Unfortunately
filesystems are designed with the assumption that every modification is
inherently stateful. (It may be possible to design a different type of
filesystem without this assumption, but every filesystem currently operates as
a sequence of commits that alter state.) So installing a library or a program
is necessarily stateful. What do you do if the program fails to install?
Trying again probably won't help: the failure is probably due to some other
missing or corrupted state. So indempotency won't help you because there's no
situation in which a retry loop would be helpful. That is, if something fails,
then whatever operation you were trying to accomplish is probably doomed
anyway (if it's automated).

I think docker is the right answer. It sidesteps the problem by letting you
create containers with guaranteed state. If you perform a sequence of steps,
and those steps succeeded once, then they'll always succeed (as long as errors
like network connectivity issues are taken into account, but you'd have to do
that anyway). EDIT: I disagree with myself. Let's say you write a program to
set up a docker container and install a web service. If at some future time
some component that the web service relies upon releases an update that
changes its API in a way that breaks the web service, then your supercool
docker autosetup script will no longer function. The only way around this is
to install known versions of _everything_ , but that's a horrible idea because
it makes security updates impossible to install.

It's a tough problem in general. Everyone agrees that hiring people to set up
and manually configure servers isn't a tenable solution. But we haven't really
agreed what should replace an intelligent human when configuring a server.

~~~
jes5199
well, the rm example is overly simple on purpose - the only thing that -f is
actually going to do that's remotely dangerous is removing files that have the
readonly bit set. I've never actually been bitten by that. In general though,
I think this pattern scales poorly - the more complicated your task is, the
more like the "force it" mode is going to be more and more dangerous.

\---

On the subject of what to do when something goes wrong: Sometimes retrying
installing a package _does_ fix the problem: if there was a network error, for
example, and you downloaded an incomplete set of files, the next time you run
it it will be fine.

If your package manager goes off the rails and gets your system into an
inconsistent state, then you have a decision to make. Is this going to happen
again? If not, just fix the stupid thing manually: there's no point in
automating a one-time task. If it is probably recurring, then, you need to
write some code to fix it (and file a bug report to your distro!). I do not
believe that there is a safe, sane way to pre-engineer your automation to fix
problems you haven't seen yet!

In the meantime maybe your automation framework stupidly tries to run the
install script every 20 minutes and reports recurring failure. The cost of
that is low.

Docker is awesome, for sure, and I'll definitely use it on my next server-side
project. It isn't a magic bullet, though - you still have to configure things,
they still have dependencies. Just, hopefully, failures are more constrained.

\---

and on the point of upgrading for security fixes: the sad reality is that even
critical fixes for security holes must be tested on a staging environment. No
upgrade is ever really, truly guaranteed to be safe. I guess if the bug is bad
enough you just shut down Production entirely until you can figure out whether
you have a fix that is compatible with everything.

~~~
thwarted
_well, the rm example is overly simple on purpose - the only thing that -f is
actually going to do that 's remotely dangerous is removing files that have
the readonly bit set._

Since you originally outlined the requirements as:

 _Take "rm" as a trivial example - when I say `rm foo.txt`, I want the file to
be gone._

then the file should be gone even if "the readonly bit" was set.

This is not only a contrived example, but a bad one, for system management. rm
is an interactive command line tool, with a user interface that is meant to
keep you from shooting yourself in the foot. rm is polite in that it checks
that the file is writable before attempting to remove it and gives a warning.
System management tools I would expect to call unlink(2) directly to remove
the file, which doesn't have a user-interface, rather than run rm.

However, the system management tool doesn't start with no knowledge of the
current state of the system, but rather one that is known (or otherwise
discoverable/manageable). And then attempt to transform the system into a
target state. They can not be expected to transform any random state into a
target state. As such, the result of unlink(2) should be reported, and the
operator should have the option of fixing up the corner cases where it is
unable to perform as desired. If you've got 100 machines and 99 of them are
able to be transformed into the target state by the system management tool and
one of them is not, this isn't a deficiency of the system management tool, but
most likely a system having diverged in some way. Only the operator can decide
if the divergence is something that can/should be handled on a continuous
basis, by changing what the tool does (forcing removal of a file that is
otherwise unable to be removed, for example), or fixing that system, after
investigation.

The other option is to only ever start with a blank slate for each machine and
built it from scratch into a known state. If anything diverges, scrap it and
start over. This is an acceptable method of attack to keep systems from
diverging, but not always the pragmatic one.

------
reitzensteinm
Personally, I use Fabric for automation, and it's got all the problems the
author says; if you get the machine into an unknown state, you're better off
just wiping it and starting fresh.

However, with the rise of virtual machines, that's a trivial operation in many
cases. Even on bare metal hardware it's not a big deal, as long as you can
tolerate a box disappearing for an hour (and if you can't, your architecture
is a ticking time bomb).

In fact, starting from a clean slate each time basically makes Fabric scripts
a declarative description of the system state at rest... if you squint hard
enough.

~~~
justizin
if you have to start from scratch, it's not declarative.

The author majorly misses out on what chef and puppet are, since both are
declarative, while chef _feels_ procedural because you are writing ruby.

These tools are based on the idea of declaring how the system should be, and a
way to get there. They also have the advantage of allowing for a design that
applies to multiple machines.

Maybe some of the ideas expressed are novel and will shape the future, but
mostly I see a lot of hand-waving and complaining about things that are Pretty
Good Ideas(tm) like FHS, which most people still don't understand.

At the end of the day, he's essentially proposing static compilation for
everything. That creates an amusing amount of attack surface, but I'll surely
find myself taking a look at Nix and maybe what the OP is talking about will
iterate toward something sensible.

That we should want something for our desktop which is a strict match for what
we want in server farms is kind of silly, IMO. See Pets v. Cattle.

~~~
reitzensteinm
_If you squint hard enough_ :)

It's not technically declarative, but in the limited circumstances for which
it makes sense, it feels declarative as you're thinking in terms of what needs
to be there without worrying at all about the current state.

While we're analogizing, it's like stocking up your fridge to an exact state.
Recipe A is:

* Check shelf for residue, clean if exists

* For each existing egg, if egg is off, throw out

* If egg carton contains yolk from broken egg, obtain new carton, transfer each egg

* While egg count less than 12, purchase and add eggs

However, if we just throw out the entire old fridge and buy a new one, we get
the recipe:

* Purchase a dozen eggs. Place on shelf.

If you can afford a new fridge each time you go shopping, I don't think it's a
bad algorithm.

------
geerlingguy
So, basically, replace yum, apt, etc. with a 'stateless package management
system'. That seems to be the gist of the argument. Puppet, Chef and Ansible
(he left out Salt and cfengine!) have little to do with the actual post, and
are only mentioned briefly in the intro.

They would all still be relevant with this new packaging system.

For some reason, this came to mind:
[https://xkcd.com/927/](https://xkcd.com/927/)

~~~
bryanlarsen
True, but:

`sudo apt-get install nginx` just works. Perhaps they're doing it "wrong", but
there are thousands of people who are making sure it just works.

I have some of the problems described in the article, but it only happens when
I can't use the package manager. It happens when I have to compile from
scratch, or move things around or mess with config files, et cetera.

For me, phoenix servers and docker are the solutions. Maybe they're not as
pretty as what he describes, but there is a solution that works.

~~~
josegonzalez
`sudo apt-get install nginx` does not just work if:

\- The repo is down \- External network doesn't work \- You are missing a
dependency not in your apt-cache \- It conflicts with another package due to a
dependency

All of which are possible and happen.

~~~
peterwwillis
Oh yeah, Debian's package management is certainly not the example i'd use of a
bulletproof system. On the other hand:

    
    
      upgradepkg --install-new /dir/*.tgz
    

Works pretty much infallibly. Nobody wants to admit it, but the most reliable
package management system is 'tar' (or cpio, really). Everything else just
introduces new points of failure. If you just want to get something installed,
there is nothing that works more effectively than completely ignoring
dependencies and metadata.

~~~
turrini
That's simple not true.

apt-get ensure that application will get the right version of a library for
its execution.

'Untaring' doesn't mean the application will work OK with their dependencies
(since there's no verification).

So anyone could do a dpkg -i --force-all *.deb. Its the same thing.

~~~
peterwwillis
Actually I believe it's more like

    
    
      dpkg -i -E -G -B --force-overwrite --force-breaks --force-conflicts --force-depends --force-depends-version *.deb
    

with one or two extra things, like installpkg/upgradepkg will prompt you for
what you want to do with config files, and adds one or two extra heuristics.
But, yes, dpkg could totally be used in a similar way as upgradepkg. It's just
more complicated so it doesn't work as well ;)

------
peterwwillis
The more complex the process you use to automate tasks, the more difficult it
is to troubleshoot and maintain, and the more impossible it is to inevitably
replace parts of it with a new system.
[https://xkcd.com/1319/](https://xkcd.com/1319/) is not just a comic, it's a
truism.

I am basically a Perl developer by trade, and have been building and
maintaining customized Linux distributions for large clusters of enterprise
machines for years. I would still rather use shell scripts to maintain it all
than Perl, or Python, or Ruby, or anything else, and would rather use a system
of 'stupid' shell scripts than invest more time in another complicated
configuration management scheme.

Why use shell? It forces you to think simpler, and it greatly encourages you
to extend existing tools rather than create your own. Even when you do create
your own tools with it, they can be incredibly simple and yet work together to
manage any aspect of a system at all. And of course, anyone can maintain it
[especially non-developers].

As an example of how incredibly dumb it can be to reinvent the wheel, i've
worked for a company that wanted a tool that could automate any task, and that
anyone could use. They ended up writing a large, clunky program with a custom
configuration format and lots of specific functions for specific tasks. It
came to the point where if I needed to get something done I would avoid it and
just write expect scripts, because expect was simpler. Could the proprietary
program have been made as simple as expect? Sure! But what the hell would be
the point of creating and maintaining something that is already done better in
an existing ages-old tool?

That said, there are certain tasks i'd rather leave to a robust configuration
management system (of which there are very few in the open source world [if
any] that contain all the functionality you need in a large org). But it would
be quite begrudgingly. The amount of times i've ripped out my hair trying to
get the thing to do what I wanted it to do while in a time and resource crunch
is not something i'd like to revisit.

~~~
haberman
> i've worked for a company that wanted a tool that could automate any task,
> and that anyone could use. They ended up writing a large, clunky program
> with a custom configuration format

Lots of people design terrible, overly complicated systems. This is an
argument for good design, not for ad hoc shell scripts for everything.

Would you write a build system in shell? Why not? Because "make" is far better
suited to the job. (Even "make" has a lot of room for improvement, but that's
a story for another day).

As another example, even Debian (known for its conservatism) decided that
systemd was a net benefit over sysvinit, where the latter could quite
accurately be described as a pile of shell scripts. Abstractions that suit the
problem domain are far superior to ad hoc imperative bags of commands.

~~~
peterwwillis
And my argument was that a simple design is good design, and in my opinion,
better than something that isn't simple.

You could definitely write a build system in shell. In my opinion, "the job"
is subjective and you shouldn't have to pick one tool for it. But I would
prefer a simple one where I can see pretty much everything it's doing up front
and not have to download source code or search through docs.

Debian picked up systemd because it was a popular decision, not because it was
better. As far as i'm aware there was nothing broken in sysvinit that systemd
fixed; there was just a vocal minority that wanted some new features and
insisted it be shipped as default.

And in defense of shell scripts, nearly every single operating system you can
think of other than Windows uses scripts to initialize (hell, even Windows has
some startup scripts), and that _is_ an abstraction that's suited to the
problem domain. It's not like for the past 50 years people have been too
stupid to come up with something better than shell scripts. The whole point of
init scripts is to let anyone be able to modify their operating system's
start-up procedures quickly and easily. If we didn't need to modify them to
suit our individual cases they'd be statically compiled C programs or assembly
(which, by the way, I have done in the past; it's not a good idea)

~~~
haberman
> You could definitely write a build system in shell. In my opinion, "the job"
> is subjective and you shouldn't have to pick one tool for it. But I would
> prefer a simple one where I can see pretty much everything it's doing up
> front and not have to download source code or search through docs.

You likewise have to read docs or the source to know what your shell is really
doing. Or search through sysvinit docs to know which shell scripts it will run
and when. I have to look up how to use shell conditionals every time, and it's
extremely easy to get it subtly wrong such that it's not doing what you think
it's doing. There are subtle distinctions between shell built-ins and programs
of the same name, and it's very unclear in some cases which is being called.
The shell is far from being a beautifully simple and transparent abstraction.
Perhaps you know it too well to be able to appreciate this.

The shell and sysvinit are abstractions over writing C programs that make
POSIX library calls directly. It's all abstractions unless you are writing
machine code directly, (and even then you have to read docs to know what the
instructions really do). So given that we're all using abstractions, we ought
to use the best ones; the one that gives us the most advantages and the fewest
disadvantages given the problem we are trying to solve. The ones that have the
greatest combination of simplicity, transparency, and expressiveness.

If you want to make sure that only one copy of a program is running, is it a
better fit to write a shell script that creates a pidfile, checks to make sure
there's a running process that matches it, and that handles a million cases
about how the two can fall out of sync? Or is it better to be able to express
directly "I want one copy of this program running" and then let the
abstraction handle the hard cases?

You might say "well the shell script is easier to debug, because I wrote it
and I understand it." Well of course, you always understand the things you
wrote yourself. But if I had to debug either a pidfile-making shell script or
a well-designed babysitter tool, I'll take the specific tool every time. If
it's mature, it's also a lot less likely to have bugs and race conditions.

> Debian picked up systemd because it was a popular decision, not because it
> was better. As far as i'm aware there was nothing broken in sysvinit that
> systemd fixed; there was just a vocal minority that wanted some new features
> and insisted it be shipped as default.

I don't think you're very well-informed about this. Your claim of a "vocal
minority" is particularly suspect, considering the decision was made by a vote
of the technical committee. Though there was disagreement, almost no one was
in favor of sysvinit. And the position paper in favor of systemd identifies
specific things that are lacking in sysvinit:
[https://wiki.debian.org/Debate/initsystem/systemd](https://wiki.debian.org/Debate/initsystem/systemd)

~~~
peterwwillis
What you're arguing for is essentially the difference between a shell script
and a more complicated shell script [some tool designed to "express directly"
what you want to do]. You ask, why wouldn't we use something more complicated
if it does exactly what we want to do? Because unless you _really need_ to do
some exact thing, it adds unnecessary complexity which leads to many
additional problems. And if you need the added functionality, you can always
add a feature or use a pre-existing tool.

Your debugging argument is bonkers. You claim shell scripting is too hard,
then say it must be easy to troubleshoot a "well-designed babysitter tool",
which requires WAY more domain-specific knowledge of debugging tools! If you
don't know how to write bash scripts, you sure as hell aren't going to have an
easy time figuring out why your package manager's borking on an update of
openssl.

Did you even read the executive summary of the position paper? "Systemd is
becoming the de facto standard" .. "Systemd represents a leap in terms of
functionality" .. "People are starting to expect this functionality [..] and
missing it could [..] make Debian lose its purpose." They only want it because
it has new features, and _people are starting to expect it_. It's a popularity
contest, and systemd won.

~~~
haberman
> What you're arguing for is essentially the difference between a shell script
> and a more complicated shell script

You seem to have a really shell-script-centric view of the world, as if the
shell is somehow a fundamental and "pure" tool, and everything else is a more
complicated version of a shell script.

What you are missing is that the shell is just another tool, and does an awful
lot behind the scenes to achieve the functionality that appears "simple" to
you. Bash is tens of thousands of lines of C and its manpage is hundreds of
thousands of words. Using the shell is not "free", complexity-wise. The shell
is a programming language, and not a particularly well-designed one at that.
Shell script programming introduces its own peculiarities, and it is known not
to scale in complexity very well.

> Your debugging argument is bonkers. You claim shell scripting is too hard

No, I claim that the shell is not a particularly simple tool. There is a
difference.

I write JIT compilers for fun. I don't shy away from things that are hard. But
my brain has only so much room for uninteresting and arbitrary facts. The very
pecular way that conditionals work in the Bourne Shell is not something my
brain can be bothered to remember.

> Did you even read the executive summary of the position paper?

You have edited out precisely the parts that contradict your position: "It
replaces the venerable SysV init with a clean and efficient design [...] It is
better than existing alternatives for all of Debian’s current use cases." Yes,
the momentum of systemd was clearly a factor, but your claim that it is only
but a popularity contest is not a conclusion that a dispassionate observer
would reach.

~~~
peterwwillis
Is it really so hard to remember these three forms of conditionals?

    
    
      if ! grep foo /some/file ; then
          run_something
      fi
      
      if [ $INTEGER -eq 0 ] ; then
          run_something_else
      fi
      
      if [ "one" = "two" ] ; then
          do_something
      fi
    

Those are the only conditionals I ever use. I don't really use conditionals in
any other way, and it's really not that complicated to see how they work.
Sure, there are more complicated forms, and forms that other shells use (what
you see above are fairly compatible, common forms of conditionals, except for
the -eq). But if you go back to the original shells and how they did
scripting, that should work for most if not all other shells today.

Also, those parts you quote don't contradict anything. It's saying systemd has
a "better design", which means it is a shinier, fancier new toy to play with.
But the paper never once points out any flaw in sysvinit. But that's obvious;
Debian had been chugging along for over over two decades with sysvinit without
any problems. If your argument is that suddenly, after 20 years, someone
realized sysvinit was some horribly flawed design that needed to be replaced,
and it just so happens that systemd came along right when they realized it, I
don't buy it. What the paper does spell out, though, is all the advantages of
systemd _for things other than system init_. Basically it says "Hey, we want
all these new features, and we need systemd to replace init for it all to
work, so please just go along with it because it's a much better design."

~~~
haberman
Here are some things that seem reasonable but don't work:

    
    
        FOO=""
        
        if [ $FOO = bar ] ; then
          echo "equal!"
        else
          echo "not equal!"
        fi
    

This errors out with:

    
    
        test.sh: line 4: [: =: unary operator expected
    

This is because the shell is based on text substitution. So once it's replaced
"$FOO" with nothing, it ceases to be an actual token, and the expansion of:

    
    
        if [ = bar ] ; then
    

...is an error. This is terrible.

One solution you sometimes see to this is:

    
    
       FOO=""
    
       if [ x$FOO = xbar ] ; then
         echo "equal!"
       else
         echo "not equal!"
       fi
    

This handles the empty string and other simple strings, but once $FOO has any
spaces or shell metacharacters in it, it will break also. This is also
terrible.

> It's saying systemd has a "better design", which means it is a shinier,
> fancier new toy to play with.

You seem dismissive of new technology. If you want to keep using CVS while
browsing the web with lynx and sending your email with pine, more power to you
(after all, graphical web browsing is just a "new feature"). But the rest of
us are moving on.

~~~
peterwwillis
It's not new technology that bothers me. Me having to do more work bothers me.
Systemd is going to make my job more difficult in terms of troubleshooting and
maintenance - way more difficult than remembering that an operator requires
two operands to evaluate.

What's really funny about systemd is I think that all its features have tons
of value, and I would definitely use them. But I also think its creators are
completely fucking batshit insane for making it mandatory to replace huge
chunks of the operating system just to get those features. You should be able
to just run systemd as a normal user process and still maintain the same level
of functionality, but for some fucked up reason somebody thought it would be a
great idea to make it a completely non-backwards-compatible non-portable
operating system requirement. It's a stupendously bad idea, and the only
reasoning anyone can come up with for _why_ they designed it that way is "It's
Advanced!" Of course, I should add the caveat that I don't care at all about
boot times, and so people who are obsessed with short boot times will find
systemd very refreshing, in the way an Apple user finds replacing their old
iPhone with a new iPhone very refreshing.

------
rbc
He left out Cfengine. That's a big gap. It's been around since 1993. He also
focused on package management and the provisioning process. I feel like there
is more to automation than that. Continuous deployment, process management and
distributed scheduling come to mind. As a plus, he does seem to get that just
using system images (like Amazon AMI's) can be pretty limited.

I think the complexity of automation is more a symptom of the problem space
than the tools. It's just a hairy problem. Computer Science seems to largely
focus on the single system. Managing "n" systems requires additional
scaffolding for mutual authentication and doing file copies between different
systems. It also requires the use of directory services (DNS, LDAP, etc…)

I like the analogy of comparing the guitar player to a symphony orchestra.
When you play the guitar alone, it's easy to improvise, because you don't need
to communicate your intent to the players around you. When a symphony does a
performance, there is a lot of coordination that needs to be done.
Improvisation is much more difficult. That is where Domen is right on target,
we can do better. Our symphony needs a better conductor.

~~~
rhys_rhaven
He left out Cfengine for the same reason everyone who has used
Puppet/Chef/Salt/Ansible leaves it out. Its utterly atrocious. Combining
global booleans as a weird sort of declarative flow control to make up for the
lack of explicit dependencies between objects that everything else has is
horrific.

Thousands and thousands of these:
(debian_6|debian_7).(role_postgres|role_postgres_exception)::

And ordering? Nah, just run it 3 times in a row. :\

~~~
rbc
It sounds like you've been using CFEngine 2. CFEngine 3 class (boolean) scope
is limited to the bundle it's called from in the bundle sequence. The bundle
sequence provides procedural flow control if you need that. Convergent
programming does take some getting used to.

~~~
thrill
CFEngine 3 cleaned up a lot of mess. The promises I write look completely
different in it - clean, understandable (even weeks later).

------
onalark
If you've ever needed version X.Y of Package Z on a system, and all of its
underlying dependencies, or newer versions than what your operating system
supports, you know exactly what Domen is talking about.

It's a good write-up. The idea of a stateless, functional, package management
system is really important in places like scientific computing, where we have
many pieces of software, relatively little funding to improve the quality of
the software, and still need to ensure that all components can be built and
easily swapped for each other.

The HashDist developers (still in early beta:
[https://github.com/hashdist/hashdist](https://github.com/hashdist/hashdist) )
inherited a few ideas from Nix, including the idea of prefix builds. The thing
about HashDist is that you can actually install it in userspace over any UNIXy
system (for now, Cygwin, OS X, and Linux), and get the exact software
configuration that somebody else was using across a different architecture.

~~~
pmahoney
> The thing about HashDist is that you can actually install it in userspace
> over any UNIXy system (for now, Cygwin, OS X, and Linux)

I've been looking into Nix (the package manager) recently, and it also can be
installed on OS X and various Linux distros (I've seen Cygwin mentioned, but I
don't think this is well supported).

If you're familiar with both, what does HashDist provide over Nix?

~~~
onalark
Sure! As a disclaimer, I'm one of the developers of HashDist. There's a longer
explanation here:
[http://hashdist.readthedocs.org/en/stable/faq.html](http://hashdist.readthedocs.org/en/stable/faq.html)
but the crux is that Nix _enforces_ purity, whereas HashDist simply _defaults_
to pure.

As an example, if you'd like to build a Nix system, you're going to need to
compile or download an independent libc. HashDist is capable of bootstrapping
an independent libc as well, but we default to using the system libc.

We choose to seat our "pure" system on top of the system compilers and libc,
but by default, install our own Python and other high-level libraries. We also
can integrate with other package managers, and there's an example branch in
our repository right now showing integration with Homebrew.

------
shaggy
The linked article is about package management, not configuration management.
Whoever set the title of this post didn't understand the point of the article.
From the comments, people seem to confuse and conflate configuration
management, job automation and package management. To run a successful
infrastructure at any scale you need all three.

~~~
pmahoney
What's the difference between a "package" and a "configuration"? At some
level, they are both sets of files, so I can imagine a tool handling both,
providing the benefit of a common interface. On the other hand, why might one
want separate tools for packages and configuration?

~~~
jzwinck
A package is a product, a cooked thing like Nginx. A configuration is
ephemeral and site-specific, a dynamic thing like ~/.bashrc. You can claim the
two are really the same, but no one else will understand what you are on about
(is he saying we should hard-code more stuff?). We don't need exactly two
categories here, but that seems the most idiomatic, and has done for decades.

Individual users may benefit from versioning their config files, but usually
not their programs. Big business might do both, or just as likely screw it up
and version code but not configs (I'm looking at you, crontab).

~~~
snowwrestler
Packages are far more ephemeral than people assume, because of the constant
flow of security patches, bug fixes, and feature updates. If you're running a
long-lived system, at some point those version numbers will matter.

I think the more useful way to think of packages and configuration are that
they are necessary compliments. An unconfigured package doesn't do what you
want, and configuration without a package doesn't do anything at all.

The theory is, they're both components of system state, so why not manage them
together?

But to me it sounds like saying that text and layout are necessary compliments
for a magazine, so why not manage them together? Because it turns out that is
a horrible idea. Anyone who does professional publishing manages them
separately.

"Do it all" tools rarely do it all well.

------
vidarh
Part of the solutions it to never update "live" machines, but to put
everything in VMs, and maintain state _outside_ of the VM images (shared
filesystems etc), and build and deploy whole new VM images.

Doing updates of any kind to a running system is unnecessarily complex when we
have all the tools to treat entire VMs/containers as build artefacts that can
be tested as a unit.

------
eranation
I'm still failing to understand what solution is there out there that handles
web application deployments (especially JVM ones) in an idempotent way.
Including pushing the WAR file, upgrading the database across multiple nodes
etc. Perhaps there are built-in solutions for Rails / Django / Node.js
applications, but I couldn't find a best practice way to do this for JVM
deployments. E.g. there is no "package" resource for Puppet that is a "Java
Web Application" that you could just ask to be in a certain versions.

How do you guys do this for Rails apps? Django apps? is this only an issue
with Java web apps?

~~~
druiid
Well, simply enough, it's kind of a difficult problem to solve. The way most
rails apps do it, is deployment with Capistrano or similar. There's also
Fabric which can interface pretty well with rails deployment as well. Honestly
the methodology I've seen under most deployment situations is far, far from
idempotent and is a bit terrible in this respect.

There are lots of tools within Rails and Capistrano that hopefully get you to
a state approaching 'idempotent' deployments, but they fairly often aren't.

------
matlock
At least some parts of this post touch on immutable infrastructure, basically
just replacing faulty systems and rebuilding them from scratch everytime you
need to change it. Relatively easy with AWS and Packer (or other cloud
providers) and super powerful. I've written about this a while ago on our
blog: [http://blog.codeship.io/2013/09/06/the-codeship-workflow-
par...](http://blog.codeship.io/2013/09/06/the-codeship-workflow-
part-4-immutable-infrastructure.html)

~~~
vidarh
It's easy on your own servers too: Containerise everything or put everything
in VMs even when you control the hardware, whether you do it with something
like OpenStack or roll your own scripts.

------
asuffield
How does this system handle shared libraries and security updates to common
components?

This is not a new idea - the "application directory" dates back to riscos as
far as I'm aware. It's been carefully examined many times over the decades,
and hasn't been widely adopted because it leads to massive duplication of
dependencies, everything in the system has to be changed to be aware of it,
and there are less painful ways to solve or avoid the same problems.

~~~
cwp
It's probably not a new idea, but it's not the same idea as the application
directory as in RiscOS.

NixOS doesn't duplicate dependencies. Instead it makes everything read-only.
Each dependency is to a specific version of the package, and everything that
uses that specific package version uses the same copy. If you want to upgrade
to a new version of a package, that implies a new version everything that
depends on it as well. (Or at least the application that you want to use the
new version.) It still uses more space than a traditional system, but not as
much as duplicating everything.

------
kzahel
I think I find myself in a minority that thinks "sudo apt-get install nginx"
is much simpler and who doesn't care about edge cases. If there's an edge
case, something is wrong with my machine and it should die.

~~~
Karunamon
Why not both? Puppet and Ansible are relatively simple. (Puppet assumes a bit
more programming experience - their config files are full of rubyisms)

They sit on top of the package manager. Ansible is more about doing commands
en masse, Puppet is more about ensuring a consistent state en masse.

What happens if you want to ensure that your server farm is all running the
same version of nginx? What if you want to ensure the configuration files are
all in a consistent state?

You can script it yourself if you really want to, but it's a solved problem at
this point. Puppet's mission in life is to notice when a server has deviated
from your specified configuration, report it, and haul the box (kicking and
screaming if necessary) back into compliance.

Manual scripting doesn't scale beyond 100 servers or so. You don't have enough
hours in the day.

~~~
dkoch
Ansible can be used as a distributed command runner but it's also a
configuration management tool like Puppet (and much more).

Its "playbooks" are by default meant to be idempotent -- you can run them over
and over ensuring that a system is in a consistent state.

------
mmcclellan
This is an insightful article for devops "teams", That said, a single devop
resource can get a hell of a long way in a homogenous Ubuntu LTS environment,
apt packaging, Ansible and Github.

I know, I know 640k will be enough for anybody, but is anybody's startup
really failing because of nginx point releases?

------
csense
I miss DOS, when there was a one-to-one correspondence between applications
and filesystem directories.

Now Windows programs want to put stuff in C:\Progra~1\APPNAME,
C:\Progra~2\APPNAME, C:\Users\Applic~1\APPNAME,
C:\Users\Local\Roaming\Proiles\AaghThisPathIsHuge, and of course dump garbage
into the Registry and your Windows directory as well. And install themselves
on your OS partition without any prompting or chance to change the target. And
you HAVE to do the click-through installation wizard because everything's
built into an EXE using some proprietary black magic, or downloaded from some
server in the cloud using keys that only the official installer has (and good
luck re-installing if the company goes out of business and the cloud server
shuts down). Whereas in the old days you could TYPE the batch file and enter
the commands yourself manually, or copy it and make changes. And God forbid
you should move anything manually -- when I copied Steam to a partition that
wasn't running out of space, it demanded to revalidate, which I couldn't do
because the Yahoo throwaway email I'd registered with had expired.
(Fortunately nobody had taken it in the meantime and I was able to re-register
it.)

I've been using Linux instead for the past years. While generally superior to
Windows, its installation procedures have their own set of problems. dpkg -S
firefox tells me that web browser shoves stuff in the following places:

    
    
        /etc/apport
        /etc/firefox
        /usr/bin
        /usr/lib/firefox
        /usr/lib/firefox-addons
        /usr/share/applications
        /usr/share/apport
        /usr/share/apport/package-hooks
        /usr/share/doc
        /usr/share/pixmaps
        /usr/share/man/man1
        /usr/share/lintian/overrides
    

I don't mean to pick on this specific application; rather, this is totally
typical behavior for many Linux packages.

Some of these directories, e.g. /usr/bin, are a real mess because EVERY
application dumps its stuff there:

    
    
        $ ls /usr/bin | wc -l
        1840
    

Much of the entire reason package managers have to exist in the first place is
to try to get a handle on this complexity.

I welcome the NixOS approach, since it's probably as close as we can get to
the one-directory-per-application ideal without requiring application changes.

~~~
asperous
You should lookup gobolinux

------
bdcravens
I've been playing with Rubber for a Rails app. It's nowhere near as capable as
Chef, but for the needs of most Rails apps deploying multiple servers and
services to AWS, it's extremely capable. I'd put it somewhere between Chef and
Heroku as far as being declarative and being magical.

------
zobzu
deterministic builds? pdebuild. mock. this exists since practically forever.

as far as the "stateless" thing, this could have been explained in a far
simple manner IMO.

1) No library deps:

"all system packages are installed in /mystuff/version/ with all their libs,
then symlinked to /usr/bin so that we have no dependencies" (that's not new
either but it never took off on linux)

2) fewer config deps "only 4 variables can change the state of a config mgmt
system's module, those are used to know if a daemon should restart for
example"

So yeah. it's actually not stateless. And hey, stateless is not necessarily
better. It's just less complicated (and potentially less flexible for the
config mgmt part).

Might be why the author took so long to explain it without being too clear.

------
gulfie
[https://www.usenix.org/legacy/publications/library/proceedin...](https://www.usenix.org/legacy/publications/library/proceedings/sec96/hollander/)

(speaking from second hand knowledge) They don't go into much of the really
interesting detail in the paper. The awesome part of all of that was that
everything an application required to function was under it's own tree, you
never had any question of providence, or if the shared libraries would work
right on the given release of the OS you might be using. And it worked from
any node on the global network. This problem has been solved, most people
didn't get the memo.

------
manish_gill
Anyone have a good introductory article about these tools (and others like
Vagrant etc)? I keep hearing about them, but so far, have been managing a
single VPS fine with ssh+git, with supervisord thrown in. Am I missing out by
not using these?

~~~
geerlingguy
I'm working on Ansible for DevOps[1]. You can download a preview, which has a
couple good chapters on setup and initial usage. Even for managing one server,
a configuration management tool is a major help; I've started using Ansible to
manage my Mac workstation as well[2] (so I can keep my two Macs in pretty much
perfect sync).

I only assume the reader has basic command-line familiarity, but I try to make
the writing approachable for both newer admins and veterans.

[1] [https://leanpub.com/ansible-for-devops](https://leanpub.com/ansible-for-
devops)

[2] [https://github.com/geerlingguy/mac-dev-
playbook](https://github.com/geerlingguy/mac-dev-playbook)

~~~
dchuk
I actually discovered your book yesterday and got very excited about it! What
is your target release date for it?

~~~
joncooper
Have you considered doing an early access program or draft editor list? I
would be interested in either if you chose to do so.

~~~
geerlingguy
I'm publishing as I go on LeanPub; I've written a little about the process so
far on Server Check.in[1]. I'll probably work on getting the book on
Amazon/Kindle before it's in its '1.0' state, but LeanPub will always have the
latest version of the book (including after the '1.0' release).

[1] [https://servercheck.in/blog/self-publishing-my-first-
technic...](https://servercheck.in/blog/self-publishing-my-first-technical-
book-leanpub)

------
renox
His example of replacing the database (stateful) by the network(stateless) for
email checking is poor: it makes the implicit supposition that the network is
as reliable as the database is.. What happen when one email is lost?

~~~
kinofcain
Well it's just supposed to illustrate an idea but in this particular example
the same thing happens as if you were using the database version and the email
with the unique code in it was lost: the user has to request another email be
sent. The difference being you don't have to go clean up all the unused tokens
out of the db.

~~~
renox
Yes, I realized just after posting this that he was talking about a specific
use case (user registration) where the user is the one making the retry in
case of failure..

So his example is OK in fact (a variation on the SYN-cookie), I was wrong.

~~~
icebraining
More generally, that process depends on _both_ the database and network, so
reliability is already the min(database, network) => network. It makes sense
to eliminate a component if you can.

------
Goladus
This looks really interesting but I don't see it as a magic bullet for
configuration management. There seem to be a lot of advantages on the package
management side but configuration management is a lot more than that.

Generally the whole point of a configuration file is to allow administrative
users to the change the behavior of the application. Treating the
configuration file as an "input" is a relatively trivial difference and
doesn't really address most of the problems admins face.

------
coherentpony
Should one really be setting LD_LIBRARY_PATH like that? I thought the
preferred way to deal with library search at run time was to rpath it in at
compile time.

~~~
jessaustin
Yeah seeing that reminds me of a consulting job I had decades ago. "We're the
customer: don't blame us when _your_ software won't run on _our_ servers!"

"OK, just set LD_LIBRARY_PATH."

------
bqe
This is why I'm building Squadron[1]. It has atomic releases, rollback, built-
in tests, and all without needing to program in a DSL.

It's in private beta right now, but if you're curious, let me know and I'll
hook you up.

[1]: [http://www.gosquadron.com](http://www.gosquadron.com)

~~~
hedwall
Does it support the major package managers? You only mention Git, tarballs and
http on the site.

~~~
bqe
At the moment it only does apt, but yum, pacman and others are coming soon.

------
contingencies
Inspired by this post (by a fellow Gentoo user, no less!) I finally published
my extended response on the same theme, which has been written over some
months:
[https://news.ycombinator.com/item?id=7384393](https://news.ycombinator.com/item?id=7384393)

------
kaivi
Talking about automating apt-get, yum and the like, is there a way to cache
frequently downloaded packages on developer machine in the same local network?

For instance, I have a bunch of disposable VMs, and I don't want them to
download the same gigabytes every time I run init-deploy-test.

~~~
unethical_ban
apt-mirror creates a full repository, and there's a tool called deb-squid-
proxy or some combo of those words that lazily caches packages.

~~~
kaivi
Oh thank you, not even sure how I missed it.

~~~
jcastro
The server piece is "squid-deb-proxy" and for the clients it's "squid-deb-
proxy-client".

------
mariusmg
On Windows i use Boxstarter or a simple powershell script that invokes
Chocolatey (must be already installed).

I had a look at Puppet/Chef.......wow those really look complicated for
something that should really be simple.

~~~
solipsism
Puppet and Chef do a lot more than install packages, which is literally all
that Chocolatey does and most of what Boxstarter does. For example, how would
you set a particular value in a configuration file with Chocolatey/Boxstarter?
How would you add a system user? How would you make changes to the firewall
settings? You can't, that's why Puppet and Chef look more complicated.

~~~
mariusmg
Powershell.

Not to mention that a lot of stuff you would do with Chef/Puppet, on Windows
are done directly from the "package" itself (create a default user with
limited rights to run under, set up IIS for web apps etc etc). And when
something fails....isn't it better to debug your code rather than a black box
?

~~~
noinsight
Actually you're right about Microsoft's solution... PowerShell.

But the technology is called Desired State Configuration which is PowerShell
based.

------
leftrightupdown
hey, i noticed this on HN so just to share my thoughts. How about
incorporating it a bit more simple, just pushing commands with some checks
like these guys do? It is a bit more low-level but automation comes only once
and should be easier to change. Here is video from their site
[http://www.youtube.com/watch?v=FBQAhsDeM-s](http://www.youtube.com/watch?v=FBQAhsDeM-s)

------
syongar
State is the entire value and utility of a computer.

~~~
leoc
Fire is the entire value and utility of, say, a pre-Industrial Revolution
forge or bakery, but it doesn't follow that therefore the more fire, or the
more uncontrolled the fire, the better:
[https://en.wikipedia.org/wiki/Great_Fire_of_London](https://en.wikipedia.org/wiki/Great_Fire_of_London)
.

------
lewaldman
Were would be Salt on that bag
([http://www.saltstack.com/](http://www.saltstack.com/))?

------
KaiserPro
This isn't stateless, the state has been moved from the package
manager/filesystem to a string held in the *INCLUDE.

This is nasty.

~~~
sparkie
That's not how it works. The Nix build system is effectively a pure function,
in that you specify all of the inputs up-front, which includes all
dependencies, and then the build system performs the build (which is _locally
effectful_ , analagous to say, using the ST monad in Haskell, or Transients in
clojure, etc). However, those local effects are discarded when we no longer
need the chroot, and the result of the build process, which we want - should
be a package that is exactly the same for all inputs. (We're not quite there
yet, but approaching this.)

------
telmich
Has anyone reading this article checked out cdist?

I like it very much, you can guess why...

------
greatsuccess
Another "functional languages make everything better", load of crap.

~~~
greatsuccess
Maybe we should just ask God to make our servers work. Since we are on the
topic of religion anyway.

------
dschiptsov
What problem does it solve besides "I am so clever and just learnt the word
'nondeterministic'?"

I would suggest another blog post about monadic (you know, type checked,
guaranteed safe) packages (uniques sets of pathnames), statically linked, each
file in a unique cryptohashed read-only mounted directory, sorry, volume.
Under unique Docker instance, of course, with its own monolithic kernel,
cryptohashed and read only.

Oh, Docker is a user space crap? No problem, we could run multiple Xens with
unique ids.

------
liveoneggs
author has a shallow or non-existent understanding of making pkgs for an
operating system, doing system's administration, or the automation tools
mentioned.

~~~
berkay
If you're going to make such a loaded, inflammatory statement, at least back
it up with some explanation on how you arrived that conclusion.

~~~
liveoneggs
I read the article. It was plenty of evidence.

~~~
copergi
Both of your posts are completely lacking in substance. The purpose of
comments is discussion. It allows people to share information and opinions,
and to learn from each other. We can not learn from comments like "that guy is
dumb". Please try to contribute to the discussion in a productive manner.

