1. We now have more decent options: Ansible [1], SaltStack [2] and all of the HashiCorp [3] stuff. All of it is a lot more sane and low barriers to entry options.
2. The recent acquisition of Ansible by RedHat [4] should be interesting. We can at least hope for a better integration, but I'd be curious to see if they get more opinionated, at least for RHEL.
Really, the best experiences I've had with this stuff is just writing plain old bash.
The problem with most of the configuration management tools is that you're inherently not trying to "configure" anything: you're either installing or upgrading stuff that's just not there, which is inherently a programatic task. Configurations for a configuration manager end up being a lot of calls out to basic bash programs anyway, except now they're written through interfaces that are only marginally different from XML, and you end up having to submit a patch to an upstream recipe to just get it to add an obscure flag. Gross. Have you forgotten that you're a programmer? Write a program, for Christ's sake!
I know exactly what you mean... we are migrating from essentially an organized collection of bash scripts + rsync to SaltStack for configuration management. I would not say it's better. It's maybe not worse either. I get annoyed that I feel like it's a sideways move, and thus not a good use of our time.
The problem with bash is if you have different operating systems spread around, like CentOS 5, 6, 7 in our case, you wind up with a whole bunch of conditions figuring out what to do for specific OS versions. That's the sort of thing you'd like one person in the open source configuration management software to solve one time for everyone to use.
There's also the non-DRYness of "a whole bunch of bash scripts." You could maybe figure out some sort of include system to avoid repetition, but it's bash... it won't be pretty.
I feel like the whole config management thing is obsoleted by containers though. We're hopeful to go straight to Docker+Mesos+Consul. The only thing you need configuration management for at that point is to build your Mesos/Docker agent hosts.
Most of the problems you're describing with bash are symptomatic of larger problems:
> The problem with bash is if you have different operating systems spread around, like CentOS 5, 6, 7 in our case, you wind up with a whole bunch of conditions figuring out what to do for specific OS versions.
This is a very poor approach to this problem in this situation. You're supporting 3 systems. It's not unreasonable to maintain 3 branches for 3 branches of the code, and with tools like `git` it's easy to backport features when it's possible without code change. This also gives you better measurement of the cost of maintaining support for legacy systems so you know when it's time to pay the cost of upgrading. And when you do upgrade, you simply stop patching the CentOS 5 branch, and you're not left with a bunch of if statements in your active codebase testing for a system that you no longer care about.
SaltStack might do better when targeting, for example, many many different Linux distros. But even then, it only solves a 99% use case, and the tools are opaque enough that fixing the 1% may take just as much time as writing more portable bash.
> There's also the non-DRYness of "a whole bunch of bash scripts." You could maybe figure out some sort of include system to avoid repetition, but it's bash... it won't be pretty.
It's really not that complicated or bad. `source` is typically all you need.
There's also a larger antipattern here, which is that if your configuration is so complicated that you're feeling pain, you're probably not curating your dependencies very well. Configuration isn't the worst way in which you're going to feel this pain: if you're letting so many dependencies into your system that you're having trouble installing them all, you're almost certainly installing a lot of bad libraries and you're going to find that you also have trouble implementing features using libraries that are poorly-designed. I'm definitely not espousing a blanket fear of NIH code, but if you don't take care on which dependencies you decide to include, you're going to end up with a lot of problems.
I agree with most of what you say. Unfortunately we have inherited a tremendous amount of technical debt that isn't as easy to pay off as it sounds. Our big apps are pretty straightforward because probably 90% of our bare-metal servers are roughly 10 core applications. That's roughly 6500 servers. It's the "long tail" of the other 500 servers with, say, 100 different applications running on them that is trickier. I probably need to do a better job justifying attention in this area.
Using saltstack, I can identify with XML-like configs and needing to patch an upstream module to add an obscure flag -- but it's worth it!
YAML seems less succinct than bash until you notice that salt allows you to abstract away distro nuances like useradd/adduser, apt-get/yum/pacman, apache2/httpd. Not a silver bullet for portability but allows your code to focus on the what, not the how.
Because salt was designed for the distributed remote execution problem domain, it outperforms any solution which just adds another layer over the legacy tools like ssh or rsync. You can patch salt to add that obscure flag, but can you patch ssh to distribute and run your latest bash script on 10,000+ nodes asynchronously?
For smaller tasks bash is just fine, but it's great to build your project on tools that you know will scale.
> can you patch ssh to distribute and run your latest bash script on 10,000+ nodes asynchronously?
Yes, quite easily, with GNU parallel. Of course, I use salt myself anyways, but it's not without its own limits. In particular, the inherent security risk of a compromised salt master keeps me up some nights.
Errr..except for the computational overhead of 10,000+ concurrent SSH connections - Salt's message queue architecture is a significant improvement there :)
1. We now have more decent options: Ansible [1], SaltStack [2] and all of the HashiCorp [3] stuff. All of it is a lot more sane and low barriers to entry options.
2. The recent acquisition of Ansible by RedHat [4] should be interesting. We can at least hope for a better integration, but I'd be curious to see if they get more opinionated, at least for RHEL.
[1]: https://ansible.com/
[2]: https://saltstack.com/
[3]: https://hashicorp.com/
[4]: https://www.redhat.com/en/about/blog/why-red-hat-acquired-an...