
Puppet vs Chef, Fight - bryanwb
http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-fight.html
======
inopinatus
I am currently migrating a largish (thousands of nodes) site away from using
Chef and back to using config files in packages, because it's simpler and has
the same effect.

Having been round the houses with cfengine, Chef, Puppet and more now I think
these tools are, overall, a poor use of time. In general they're 99% used as
config blasters: Yet Another Way To Put Files Onto A Computer.

Turns out, packaging systems already did that and have better dependency
analysis.

This also helps to match the lifecycle of configuration management objects
with that of the components they configure. I've seen far too many sites that
had one big hairball of a Chef/Puppet repository that tried to service the
needs of multiple conflicting application releases.

The final nail in the Chef coffin is that it encourages parameterisation of
config files rather than configuration by convention, which is simpler and
less prone to production-environment gotchas.

Everything else they do can be replaced by a very small shell script.

~~~
socratic
Can you go into a bit more detail about setting up nodes without something
like Chef/Puppet?

What tools _do_ you end up using? Some sort of custom apt repository plus some
shell scripts? What does it look like?

~~~
inopinatus
We have some Debian and some CentOS nodes, and yes it's an apt & yum
repository. The repo itself is subdivided, debian-style, into "unstable",
"testing" and "stable" which matches the states of components in our
continuous delivery pipeline.

So we do per-commit integration testing with "unstable", promote into
acceptance & functional & regression testing (and showcases etc) with
"testing" and then promote into "stable" for production. The actual promotion
is done with some very small shell scripts off a CI server; the installation
is managed via rundeck.

The configuration packages are per-node role dependencies and we try to keep
granularity large - e.g. there's a configuration package for a core Java
webapp that includes the application's own resources & nginx & jetty
configuration bits.

For per-environment stuff (like database passwords and external integration
endpoints) we just inject a (quite small) yml file to each server for the
config packages to find. But configuration by convention is preferred so we
also manage the DNS carefully by role (again, from rundeck) so that services
are at well-known unqualified label names.

Finally, the per-environment spec itself (i.e. that the rundeck scripts look
at) is in a cheesy cvs tree. I keep meaning to move that to git.

The base images themselves are stock virtual machine templates (AMIs for EC2
testing/dev) and the config packages & rundeck take it from there. The first
step is to install a basic platform package that has lots of dependencies for
our common tools, libs and needs, the the role-specific config package does
the rest.

~~~
grandalf
Very interesting comments. I have lately been trying to choose between puppet
and chef and I think I had a hunch that there was a better/cleaner way.

What do you recommend for managing the running state of processes, etc?

~~~
inopinatus
I prefer to use whatever's native to the platform. On Debian that means start-
stop-daemon; on CentOS the init.d functions.

It's a whole "my house, my rules" strategy. If you play nice with the
underlying platform, it'll play nice with you. The downside is that you do
have two packaging models to build for. I know that abstracting this is one of
the motivations for Chef et al. I have yet to see a usage where this was worth
the opportunity cost.

~~~
grandalf
Interesting. Thoughts on upstart? BTW do you use anything along the lines of
nagios, god, monit, etc.?

------
spez
Our experience with puppet at Hipmunk was dreadful. The configuration language
is clearly designed to cause maximum pain.

We have since switch to Fabric <http://fabfile.org>, and are much happier.

~~~
bryanwb
I have written tons of fabfiles and Fabric is a great project. However it
isn't intended to managed system state. It is great for code deployment but
not suited to configuration management.

~~~
spez
I would argue that Puppet is also not suited to configuration management.

~~~
adient
You can argue whatever you want, but Puppet's job is to enforce system state
whereas Fabric simply allows programmatic use of SSH. Fabric and Puppet are
completely separate projects with very different goals.

~~~
mattdeboard
I've written loads of fabfiles and a subclass of Task a few times. I've also
wrestled with Puppet and Chef and been resoundly defeated by both.

Fabric does what we need it to do for now for provisioning & deployment. And
frankly you can make it do whatever you want. "Programmatic use of SSH" isn't
something to be scoffed at; it's simple yes but it also lets you do whatever
the hell you want very simply.

~~~
adient
Yes, of course. Nothing is wrong with using Fabric. It's a clear step above
trying to use shell scripts to manage your servers and deployments. But what I
am seeing across several comments is that tools like Puppet and Chef are
clearly misunderstood by many developers. As a sysadmin myself, I would say
that there is definitely a larger picture to the use of Puppet/Chef than what
developers are looking for, and thus something like Fabric is much more
suitable.

------
techscruggs
I use Chef, but without the server (chef-solo). I have a hard time stomaching
the idea of dealing with a Merb application (an abandoned framework) that has
some rather complex dependencies (couchdb & rabbitmq). That being said, plenty
of peers have told me that they have not experienced any problems with running
a Chef server.

Maybe this means I should just fork up the cash for hosted Chef, but I don't
see it as a 100$ a month value (I currently have 10 nodes and treat them all
as ephemeral).

I do wonder why this is never brought up. Is it not seen as a red flag to
anyone else that the core of Chef is built on Merb?

~~~
Woost
Just as a warning: If you/anyone decides to self host a chef server, you
_have_ to get the chef-server cookbook (or your couchdb will never get
compacted) and you can't be running ruby 1.9.1 (or the server will have ~1
week uptime before crashing randomly)

As for the core being built on merb. I don't really notice it. You'll probably
never have a reason to go into the chef-server internals to change something.
Besides that, it's not like it has to be serving 30k requests per second or
anything.

Otherwise, the Issues I've run into with chef-server: it was a little annoying
to get everything installed and playing together. Couchdb + solr are very
temperamental. After everything was running and stabilized(1.9.1 bad!) I've
had no issues with it.

~~~
techscruggs
What version of chef server are you using? It appears to have been fixed a
while again: <http://tickets.opscode.com/browse/CHEF-489> .

Perhaps your issue is not tied to your ruby version:
<http://tickets.opscode.com/browse/CHEF-920>

~~~
Woost
The problem wasn't ruby 1.9, but ruby 1.9.1 specifically. I upgraded ruby to
1.9.2 and the stability issues went away. On 1.9.1 the server would randomly
hang and require a kill -9+manual restart.

------
socratic
Am I totally missing some obvious Chef documentation? The entirety of the
documentation when I last looked seemed to be the wiki + the one 50-page
O'Reilly book.

I ended up literally printing out the wiki. (And the wiki seemed to be in a
state of pretty extreme flux and/or disagreement with what blog posts
suggested was best practice.)

The business model of the companies promoting Puppet and Chef seems to be to
charge for support and/or hosted services. Which is fine. But is it leading to
abysmal documentation?

~~~
nigelk
"The business model of the companies promoting Puppet and Chef seems to be to
charge for support and/or hosted services. Which is fine. But is it leading to
abysmal documentation?"

This is one of the reasons we're moving Puppet Labs from being a support
company to a product company, and not for hosted services.

If your bread and butter comes in from support, you have no incentive to
actually make your product easier to use. There are plenty of enterprise-y
software companies who make lots of money operating like this, but that's not
the sort of company I want to work for.

I'm quite proud of the rapid improvement we've made on the Puppet Docs over
the last year since we hired NickF, our most excellent tech writer:

<http://docs.puppetlabs.com/>

I particularly like the solution focused docs he's done, as opposed to the dry
reference material that presupposes a lot of knowledge.

<http://docs.puppetlabs.com/learning/>

Anyway, just wanted to point out that that's not our business model.

(Product Manager at Puppet Labs)

~~~
dredmorbius
Nigel, that's an excellent move.

The one persistent thread I've seen with "support-based models", notably Red
Hat, is that ... the tools leave you requiring support, and yet the support
still sucks (in large part because the support engineers have to use the same
crappy tools you do).

I'm not going to pretend I've got the Free Software business model sorted out.
But I _have_ noticed that project goals and incentives do line up with
results. Debian has a social contract, a constitution, and policy, _and the
tools to support these_. From an ease-of-use / ease-of-maintenance
perspective, Debian (and its derivatives) blow RPM-based systems out of the
water. And to be clear: the distinction is much less the packaging formats
(though that's part of it) and far more the use and management of these tools.

I think there's someone who died recently who made the end-user experience
key. While his organization struggled at times, it's emerged in recent years
as an absolute behemoth which suggests ultimately that the alignment he chose
is better than that of a principle rival.

~~~
nigelk
Thanks for the positive words.

I'm not going to pretend we've got the Free Software business model perfectly
sorted either.

We're producing proprietary software as well as developing/curating open
source software projects, and there is a balancing act involved there.

Not to sound too bandwagon-jumpy (I've found the recent hyperbolic public
outpouring of grief somewhat distasteful) but I care about user experience and
documentation, and I've always believed that focusing on the end user will
lead you in the right direction.

[http://puppetlabs.com/blog/what-is-user-experience-in-
puppet...](http://puppetlabs.com/blog/what-is-user-experience-in-puppet/)

We're looking for more UX staff, and have an open UX Designer role right now
actually...

[http://puppetlabs.jobscore.com/jobs/puppetlabs/ux-
designer/b...](http://puppetlabs.jobscore.com/jobs/puppetlabs/ux-
designer/bRFWXw3Mar4juDeJe4bk1X)

------
viraptor
Something that wasn't visible in the article: Not that Puppet is extremely
better here, but I really hate how much manual tweaking Chef requires after
installing from a package and the experience in running it needed. You cannot
easily scale it (indexed database pretty much needs to stay in one place),
couchdb stops replication at random and it seems that chef doesn't deal well
with document versioning in general since I keep running into a situation
where the cookbook is not on the list after uploading, or a visible cookbook
cannot be deleted. It also sometimes requires reindexing for some unknown
reason, or the attributes don't work properly in search.

I prefer to work with Chef much more than Puppet (as a user/developer), but I
wouldn't want to be responsible for running Chef cluster itself. There are too
many different elements to learn and take care of. Unfortunately official
hosted service is very expensive, so there's no good alternative once you need
many nodes.

------
moe
I'm running both chef and puppet in production, so I will add my (slightly
cynic) comparison here:

When comparing Chef to puppet then Chef comes out as the pragmatist. Once up
and running most common every-day tasks are less painful in chef. Apart from
that its main advantage over puppet (to me) is that it allows to semi-sanely
manage transient hosts (cloud/EC2) that enter/exit a cluster ad hoc. Puppet
can also do that in theory, in practice you'd rather want to fork your eyes
out with a spoon.

However, the pragmatism comes at a price: The chef-implementation is an
absolute and unmitigated disaster.

You'll spend quite a bit of quality time initially to get the six dozen
components to play ball and to fix up basics that shouldn't need fixing (i.e.
you'll want to ensure that _everything_ is under version control and not just
the parts that chef deems worth versioning).

Over the first couple months you'll also see the odd server-crash while you
figure out the peculiarities of your particular installation. Chef is _very_
heavy on dependencies and the exact software-versions depend on when and how
you install (pkgs vs source-code).

However, once you're over that hump and if you're not too worried about
standing on the shoulders of 'a one-eyed amongst the blind' then the whole
cookbooks/roles/runlists arrangement is quite comfortable to work with.

Just don't expect features like dry-run, real idempotency or clean dependency
tracking that some would consider "basic" for such a tool. Also don't expect a
security-model at all; to my knowledge all hosts that are authorized to talk
to a chef-server can see all cookbooks and databags on that server.

If you care a lot about those latter minor quibbles then perhaps Puppet might
be more your thing.

Puppet is conceptually much cleaner (night/day difference), which sadly and
ironically is also its biggest drawback; they took it too far. Puppet made a
bad decision early on by inventing their own language. This decision will be
your personal ball on a chain for the lifetime of your puppet deployment.

But, is it really that bad? Well. Yes.

After the initial (steep) learning curve there's only a small plateau of
reward before you begin to run into the more subtle issues. The most commonly
heard complaints about the language are the ass-backwards class/variable-
inheritance and the blurry (and effectively undocumented) divide between what
should go into the manifests (sorta like chefs "cookbooks") and what into a
storage layer called "extdata" (sorta somewhat like chefs "databags"). But
rest assured, there's plenty more, I don't want to spoil it all at once here.

So, yes, you _will_ hate puppet every time you have to make a complex change.

Yet for some it might still be worth it, here's some of my reasons: Once you
finally have something up and running puppet feels much more predictable and
"solid" than chef. You can actually dry-run and test (most) changes before
rolling them out. Puppet will provide meaningful error messages in most
situations (unlike the esoteric chef stack-traces). The puppet daemon is just
that; _one_ daemon (unlike the conglomerate of moving parts that comprises a
chef deployment). Generally speaking there is much less "magic" in puppet than
in chef. You will almost always know precisely what went wrong - a pretty
important attribute that chef unfortunately doesn't share.

Oh, and no least: the puppet documentation is heads and shoulders above chef
(although the latter has improved recently).

So, if you're in the market, good luck making your choice. I'm not making a
recommendation here because, quite frankly, I wouldn't recommend either to
anyone other than my worst enemy. ;-)

~~~
reinhardt
From a complete outsider's point of view, it sounds as if the state of the art
SCM tools are at the pre-2005 level of VCS tools. Chef and Puppet are the SCM
equivalents of CVS or Subversion. The Git/Mercurial equivalents (in terms of
power) have yet to appear. Does this analogy make sense?

~~~
Goladus
Not bad but rather awkward I think. Source control is a comparatively specific
problem. System configuration management is a much broader category, and
competes indirectly with a variety of existing tools, like fabric or even make
and bash.

------
lusis
The correct answer to this entire question is "yes". Use whichever tool
encourages you to adopt proper configuration management.

Yes, there's a gap that the current crop of CM tools don't address (and why we
see new versions of capistrano clones with a bit of system management
sprinkled in) but the three major tools right now - Puppet, Chef and CFengine
are all at a state where they address 95% of the use cases for system
management and automation.

~~~
lobster_johnson
What do you consider to be missing in the current configuration management
tools?

~~~
lusis
It's not so much what I consider to be missing as what others don't seem to be
finding in them that encourages them to bolt on functionality into
"incompatible" tools:

[http://blog.lusis.org/blog/2011/08/22/the-configuration-
mana...](http://blog.lusis.org/blog/2011/08/22/the-configuration-management-
divide/)

------
keithlard
A little while ago I wrote a piece comparing Puppet vs Chef on 10 different
criteria and concluded that Puppet wins on all of them:
<http://bitfieldconsulting.com/puppet-vs-chef>

It stimulated a very interesting discussion in comments in which some of the
leading lights from both communities (and there is some overlap) took part.

~~~
lusis
A little while ago is a bit of a stretch. BOTH communities have moved far and
beyond the state of things in that article.

The fact of the matter is that X vs. Y is ENTIRELY subjective. It isn't a
sporting event where there's a clear cut way to call a winner.

Some people like Puppet. Some people like Chef. Anyone trying to decide
between ANY software - whether two competing client libraries or something as
critical as configuration management owes it to themselves and the company to
try all the options. The one that's best is the one that encourages you to
adopt configuration management.

------
pwelch
This is a good write up. As someone who just recently started messing with
Chef I found comparison articles non-existent. Though I have already put in
several weeks with Chef and will probably stick with it I think Puppet looks
like a great product as well. I hope both of these projects continue to gain
users. It would be great to see them both compete/innovate well into the
future. With that said I would like to see Chef's documentation increase and
the merb application replaced by something else as someone else suggested. In
the end they are both great tools and if you are sys admin it wouldn't hurt to
learn one really well and become familiar with the other just like working
with different Linux distros such as RHEL/Ubuntu only increases your sys admin
skills.

------
rizumu
I've been happy with Kokki and Fabric, no complaints yet:
<https://github.com/samuel/kokki> If I outgrow it I'll be happy to switch to
chef or puppet, but it is still convenient to keep everything in Python.

------
kbd
While I often see "Puppet vs Chef", I never see Bcfg2 mentioned. Is its low
mindshare related to its quality, it's exclusion from the set of Ruby-based
config management tools, its origin in the government space, or its terrible
name?

~~~
agj
It's probably the XML.

Not only is Bcfg2's explicit configuration structure unpleasant to work with,
the heart of it is in XML. I tried to use Bcfg2 and was easily put off by the
required structure. The fact that Bcfg2 stems from the science/math community
was actually the only thing that kept from overlooking it completely.

------
leoc
How does the free version of CFEngine <http://cfengine.com/> compare to these
two?

~~~
pwelch
I'm curious about this as well. Does anyone have any experience with this? It
would be interesting to hear someone's opinion since CFEngine seems to have
been around for awhile.

~~~
Goladus
I have used cfengine2 extensively, but not cfengine3. I've only read about
cfengine3.

cfengine2 is a simple and effective mechanism for managing systems. However,
cfengine2 has some issues. It has limited facilities for abstraction,
resulting in a lot of code duplication. It has no built-in template engine,
meaning that if you need to assemble a file based on various conditions you
have to cobble it together with pattern matching and line-insertion. It's
pretty easy to screw this up and lose idempotency. There's a 30-package-
install limit per cfagent run, which fails silently. This is very annoying and
requires some hacks to circumvent.

Also, in cfengine2 you have a configuration file that's basically <classname>
: <list of hosts...>. If you approach more in terms of <hostname>: <list of
classes>, it's a minor inconvenience.

Cfengine3 addresses some of these issues, but I haven't used it much so can't
really give it a fair review.

------
ajdecon
This article came at an excellent time: I've recently been setting up test
deployments of both Puppet and Chef for potential HPC use, and this article
confirms a lot of my impressions. In particular, that Puppet has much better
documentation (especially for doing the initial setup) and a more sysadmin-
friendly language, but that Chef makes it easier to carry out complex tasks.

Despite some cool things I've been able to do in Chef, Puppet is probably
going to end up the winner, if for no other reason than it having much much
better support for RHEL and clones (ie Scientific Linux). Chef was extremely
unimpressive there in many ways, and anything related to deployment where the
wiki includes the line "RPM installation via ELFF has been deprecated" is
going to be a non-starter in the HPC world. (Or doesn't, anymore... any
mention of RPM seems to have been scrubbed now.)

We're probably only going to use it on infrastructure/head nodes though: our
compute nodes are provisioned on ramdisk statelessly using Warewulf, and are
easier to reconfigure with a reboot/redeploy than via CM.

(edit to clarify)

------
rmoriz
It's sad that chef differs so much depending on server/solo usage:

Without chef-server using so called "chef-solo" lacks a lot of good things:

    
    
         + (remote) bootstrapping + distribution (like "little-chef"; non-official)
         + data bags (third party cookbook only; non-official)
         + search (e.g. on a central file system based data-base which gets dumped into a json that will be uploaded on bootstrap/update time (push))
    

As others stated, running your own chef-server (even external/paying for it)
brings a new risc factor into your provisioning business and if you depend on
data bags and search you'll most likely not be able to bootstrap new app
servers when your chef server is down. This is a huge SPOF.

------
atsaloli
For those getting started with Configuration Management, I would recommmend
"Getting Started with Configuration Management" by Cory Lueninghoener (in
April 2011 issue of USENIX ;login: magazine):
[http://www.usenix.org/publications/login/2011-04/openpdfs/Lu...](http://www.usenix.org/publications/login/2011-04/openpdfs/Lueninghoener.pdf)

and my own report from Configuration Management Summit last year:
[http://www.usenix.org/publications/login/2010-10/openpdfs/Co...](http://www.usenix.org/publications/login/2010-10/openpdfs/ConfigMgt10reports.pdf)

------
knodi
I haven't used Puppet but I have had the misfortune of using Chef (1k+ node
deployment). But for a free alternative its not bad, it gets the job done sure
we'll have to sacrifice or work around it but in the end you can do what you
need to do with it. Yes, its not RightScale but thats why RightScale is not
free.

I haven't tried out scalr my self but I hear it described as an alternative to
RightScale with 90% of the features at 10% of the cost.

If anyone here has used scalr please enlighten us of on its current state.

<https://scalr.net/>

------
maratd
> If you aren't yet using Puppet or Chef for managing your *nix
> infrastructure, you should seriously consider it.

Why? I prefer to simply write my own bash deployment scripts. Seems easier.

~~~
troels
Depends on what you mean by deployment. We use puppet for managing our os,
packages and basic file structure. We use home-grown bash scripts for
deploying new versions of applications.

For keeping track of os-level packages and the like, I find Puppet much better
than something I could build myself. It doesn't just take care of the initial
setup, but also handles incremental changes. And it does so in a largely
declarative language. It all guarantees that the entire installation is
documented and replicable.

~~~
techscruggs
Agreed on all points.

Guaranteeing the entire installation is "documented and replicable" is huge.
Chef/Puppet can become a large part of you disaster recovery plan and
increases the "bus number" in ops.

The value in this is immediately apparent after the first time you have had to
rebuild a server that you inherited.

------
bostonvaulter2
I've been using puppet recently, one thing I wish for is an easier way to pull
in other puppet's users modules. I've looked at puppetforge some but I balk at
actually downloading the files to run locally. Is chef better in this regard?

------
jarin
I use Puppet, but primarily because of Moonshine, a Rails plugin that ties
Puppet together with Capistrano and Rake, and has a nice plugin system.

------
olegp
Does anybody else think there's room for a lightweight alternative to these
two projects written in server side JavaScript?

~~~
burgerbrain
Why should the language matter? Functionality should be the feature.

~~~
olegp
JavaScript is more widely accessible than Ruby or a DSL. So given that the
point is all about creating recipes, the language is a defining feature.

From what I can tell both Puppet and Chef seem a little over-engineered, so I
feel there's a niche for a simple (solo only) tool out there. By using
something like Node it would be possible to a) run with a much smaller memory
footprint b) have a more event driven architecture and c) run multiple
downloads or other setup related tasks in parallel.

~~~
count
What javascript engine ships on a linux server today? Ruby and Python are both
packaged in by default on basically everything, V8 and other javascript
engines are not.

Additionally, javascript is NOT more widely accessible - if anything, learning
non-DOM-manipulated javascript is INCREDIBLY painful for a beginner.

Remember, these tools are for system administration and configuration
management, not for writing applications.

~~~
focusaurus
While javascript engines are generally not shipped with OSes these days, it's
moot because chef bundles its own ruby and ignores the ancient one in the OS.
I suspect that's the case with python-based packages as well. I know HP Server
Automation bundles its own python. It's easier to support a consistent python
version across umpteen platforms than to debug your code in all the varying
versions across so many distros and distro releases. Maybe Fabric can use the
OS's python, but that's about the only example I know.

------
EwanG
AM I the only one who saw the title and assumed this was going to be the
Muppet's Swedish Chef vs Emeril or someone similar?

