
Why Chef? - bryanwb
http://devopsanywhere.blogspot.com/2011/10/why-chef.html
======
apinstein
I am a co-founder of a small SaaS company. We recently decided to make the
investment of upgrading our infrastructure setup process from "Hey David go do
that" to being 100% chef-based.

I managed the process, and consider it a success. However, here are some
points I would make:

1\. It took a long time. Let's be generous and say it took 2-3 man-months of
time to set up 4-5 different projects and roles. This was probably 10-20x what
it would've taken to set up the servers directly. Why? Learning curve with
chef for both our programmer and sysadmin. Figuring out how to make config
changes automatable and idempotent.

2\. The scale you get from chef is bigger than managing production
infrastructure. We now use chef for not only production deployment, but also
dev. Once paired with Vagrant, we are able to get new devs up with a complete
stack in about 10m of keyboard time. If we need to upgrade to some new version
of something, only one person has to deal with the sysadmin; everyone else can
just update their box.

3\. I think it will save money in the long-term. A good sysadmin is $100/hr+.
Unfortunately you have to pay that rate whether they're doing architecture,
security review, or just editing text files. With chef, a non-sysadmin
resource can generate recipes with just architectural advice and review from a
sysadmin. This is much more efficient, especially for small shops where a
sysadmin is an expensive and not immediately available resource.

~~~
tptacek
A good sysadmin costs $200k/year? Or do you have a contract sysadmin? Because
that number sounds way, way, way high.

(We recently hired a sysadmin "+").

~~~
apinstein
We've been in business for about 8 years and use about ~100 sysadmin hours a
year.

I interviewed several sysadmins over the years and no one that I deemed
competent charged less than $100/hour.

I did find people charging as low as $50/hour but they didn't seem that great,
or that reliable, or that available.

For me as well, you pretty much need to trust your sysadmin more than almost
anyone else in your company (except people that can sign checks).

Trust comes at a premium.

I will yield that I don't like working with substandard people. I hate having
to manage people and am willing to pay a premium for people I trust to work on
the right things with the right skills in a timely manner. Besides, it doesn't
scale. One of my business goals is to never have middle management.

~~~
tptacek
You'd be more convincing if you didn't imply that anyone who worked for less
than $100/hr was "substandard". For full-time salaried, I assure you, $200k/yr
is not the going rate for sysadmin.

~~~
nestlequ1k
You cant directly equate contract hourly rate with yearly salary. If you don't
have the budget to hire a 100k a yr salary, you'll have to pay contract rates
which could be over 100/hr

~~~
tptacek
I buy that the valley is so hot right now that a sysadmin commands $100k/yr
there, but they don't in Chicago, Seattle, or New York.

I'm not looking to argue so much as to inject some more data into the price
point that was casually dropped on this thread earlier. I do not think the
other guy overpaid for sysadmin; if he's got an amazing admin, great! I can
see paying a premium for that.

Another point I'd like to raise is, if you're paying $100k/hr for sysadmin,
and using them frequently, contracting instead of fulltiming sysadmin might be
penny-wise-pound-foolish. But maybe not, if you're only paying $10,000/yr in
sysadmin. We've used over 100 hours of admin in just the last couple weeks. A
great hire; one we made "37signals-style", after realizing that doing all the
sysadmin chores ourselves was subtly making us all miserable and ineffective.

Believe me, I know the difference between contracting rates and salary. ;)

------
codypo
I agree with the author that Chef is an incredibly powerful tool and that it
has numerous benefits over plain old bash. However, I'm reluctant to actually
use it because of its dependencies. Chef relies on CouchDB, RabbitMQ, and
Solr, and all of those have non-trivial dependencies as well. Then, with a
stack like that, I worry about the overheard involved.

FWIW, Puppet's dependencies are much simpler. I don't know much about Chef vs
Puppet, but I can say that from an installation and dependency maintenance
POV, Puppet wins.

~~~
jtimberman
(disclosure, I work for Opscode.)

To be clear, those dependencies are required for running your own open source
chef server.

You can use chef without the server, as chef-solo, or let Opscode run it for
you in the form of Opscode Hosted Chef. We do make it easy to install Chef
Server through Chef Solo itself, or with our Ubuntu/Debian apt repository.

~~~
moe
I never understood what kind of person/company would trust a hosted chef.

The chef databags/cookbooks tend to contain rather sensitive information (ssh-
keys, passwords). Handing all that stuff over to a third-party borders on
criminal negligence to me.

~~~
jethroalias97
So... should nobody use ec2? Or any host for that matter? Sooner or later
you're going to have to trust some third-party.

~~~
moe
There's a difference between trusting someone with your physical hardware and
handing them your credentials on a silver plate.

There's also a pretty harsh difference between the security practices at
Amazon and the practices that Opscode displays in their OSS-code.

------
gpapilion
I have three fundamental issues with Chef;

1\. The node object ends up being two large, which leads to memory issues when
a search returns more that 200 nodes (800MB of memory).

2\. Chef discourages declarative configuration.

3\. Chef lacks a remote trigger mechanism.

Issue one starts to kill you once you have a large number of nodes. Dedicating
a quater of the available memory to configuration management seem like a poor
financial choice. We've come up with work arounds at my company(generate files
centrally, and distribute with chef remote_file syntax), but I still feel they
are hacky.

Issue two is more serious. Reindexing on chef only occurs after a node has
submitted its node object back to the chef server. This results in incomplete
searches until a node successfully completes a run. If you wish to remove a
node from a particular role or attribute from a host, you may have a hard time
doing so until the next chef run completes.

Problem three is really a result of the expense of running chef. If the memory
and CPU costs were lower, there wouldn't be any real issues running Chef more
frequently. Some changes I need to go out immediately, some don't matter. I
end up back in the world of the SSH loop too often with Chef.

~~~
jtimberman
1\. We are working on making the search more performance and use less memory.

2\. Chef definitely does not discourage declarative configuration. Chef
recipes include declarative resource for configuring your infrastructure.
Since recipes _are_ an internal ruby dsl, there may be nondeclarative code in
them.

3\. Chef itself doesn't have a remote trigger mechanism because the. Her run
is all about configuring the local node. Nothing prevents you from using the
ruby language in a recipe to hook up some kind of remote trigger though.
People in the chef community are doing this with projects like Noah and Pylon.

Github.com/lusis/Noah Github.com/fujin/pylon

~~~
gpapilion
1\. I understand that, and have heard that from you guys several times. The
issue is the deserialization from JSON. The monkey patch solution I've see so
far stores less data, essentially white listing the attributes for search. It
sort of destroys the value of search.

2\. I should be more specific. Generally chef relies on the information that
ohai provides, not with information enumerated by the administrator. There is
a general assumption that the systems are properly configured, and chef is
only furthering that, since the hosts provide most of the configuration
details. (Yes, you could do stuff with data bags to address this issue.)

3\. Thanks for the links. I've not seen those projects previously. I've solved
the issue for myself using a much lighter weight solution.

~~~
jtimberman
1\. Yeah, the stop-gap "solution" is to white list a number of attributes in
order to reduce the data set, because in the most common use case we see,
there's only a few attributes that people _actually_ care about in the chef-
client context. The node's run list, its IP address or FQDN. Really. That is
the most common. For like, 90%+ of the use cases out there. Everyone has a
unique snowflake and thats cool, but really, not that much.

2\. There's no assumption that systems are correctly configured other than
they start from a baseline configuration in the most common use case. We have
worked with several customers managing existing infrastructures of running
systems that had an unknown baseline and Chef was able to automated the pieces
they cared about.

Again with "most common use case."

------
adient
I think this post should be titled Why Configuration Management?, with a
subtext of using Chef as an example. The main points the author is making are
true of CFEngine, Puppet, Chef, and other config management software. The
question of Why Chef? can be answered very simply: because the author likes
it. Which is a perfectly valid way to choose your tool chain, assuming the
technical requirements are met.

~~~
bryanwb
I wrote about that it the previous post, see her
[http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-
fi...](http://devopsanywhere.blogspot.com/2011/10/puppet-vs-chef-fight.html)

------
nona
Two reasons I'm wary of Chef wrt Puppet

1) the declarative vs imperative aspect

2) Chef's heavy dependencies

As a Ruby developer, I like Chef's Ruby DSL; but I somehow feel that its
imperative DSL will lead to something similar to Bash Hell. I'd like to read
more about the declarative properties of Puppet vs the imperative way of Chef,
and why one should prefer one over the other.

Secondly, like some commenters before mention, Chef's dependencies seem quite
excessive.

~~~
parasubvert
Chef is becoming more declarative as it grows.

The main difference with Chef is that it has imperative structure (i.e. ruby-
based scripting) to fall back on, whereas Puppet forces you to go out to a
script if you want to be imperative.

In the long run, it's arguable that a strict declarative structure has more
interesting properties when dealing with query of status & handling changes or
drifts rather than "stamping out a server".

Puppet has been great for me to detect and correct drift, for example, all the
while ensuring pre-requisites are executed in the proper order. The tradeoff
is that you have to think through the various dependencies at a detailed
level, which can be difficult. An extreme analogy would be programming with a
logic language vs. an imperative language.

My experience has been Puppet is growing in popularity with enterprises, not
just web companies, and it seems Puppet Labs is targeting this audience more
than Opscode is, comparing customer lists. I'm not quite sure why this is -
could be attitude - Chef is more about "get it done now", Puppet more about
"get it right for the long haul", but even that is a caricature.

~~~
lkanies
(disclaimer - founder of Puppet)

We at Puppet Labs have always focused on building tools that anyone can use,
not just the best hackers in the world. We've got some of the best sysadmins
working with us, but we punish them by making them write software that even
people who aren't the best sysadmins can use.

So yeah, we get a ton of enterprise adoption as a result, but we've also got a
ton of web companies using Puppet. It's true that in the Rails web startup,
we're not always the number 1 choice, though. :)

------
grandalf
For < 50 servers, can anyone comment on my choice of fabric and cuisine as an
alternative to chef?

~~~
JoachimSchipper
The number of different configurations is far more important than the number
of boxes, no? If you have 1000 completely identical boxes (e.g. big compute
cluster), a shell script that sets up a freshly installed box to your
requirements is quite possibly sufficient.

------
kim0
I've been working with juju <https://juju.ubuntu.com/> (a new Ubuntu server
tool) and been very happy with it. It does not directly compete with
puppet/chef for enforcing a server configuration however. Juju operates at the
higher level of "services" that can be deployed to a cloud, to hardware, or to
local LXC containers! It's basically like "apt-get" except for servers. The
dev team hangs out at #juju on freenode irc

------
leandrod
I cringe at CouchDB. Gimme good, old PostgreSQL.

