1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.
2. It attempts to move some configuration management primitives into the build pipeline so that the deployment config has enough flexibility for real-world use.
3. It intends to replace single host process supervision with a distributed model based on gossip.
This project is most definitely ambitious, and I'm not sure the whole goal is realizable but some aspects are particularly interesting.
1. The build process seems quite useful for building containers, especially if it gets some traction. The main drawback of nix is that no one knows how to use it. If this gets the backing of the hordes of chef-aware system administrators, we may have a good solution for isolated repeatable builds, especially if they focus on deterministic reproducible builds as well.
2. Distributed systems are notoriously hard to configure in a general way. It seems like every company has their own magic combination of config management, monitoring, and scripting for keeping systems like zookeeper running. Its pretty clear that we don't have the right primitives for sharing this work. I don't think a container management system has the right primiteves for this either, because I don't think you can solve for all distributed systems in a generic way. You really need custom logic for the particular software you are trying to distribute. I'm not convinced that a gossip-based process-supervisor is the best approach for this, but it could give us a place to start collaborating on these primitives.
Except the builds aren't at all repeatable in a meaningful way, judging by the example code on the home page. The 'do_build' function runs 'npm install'. This means that 1) the build container has network access (i.e. it's nondeterministic by definition) and 2) the declared dependencies do not fully describe the dependency graph. So, I really don't see how much it could possibly borrow from Nix if it threw out the most important part of it.
Nix also has similar generators for rubygems, python, haskell and some other languages.
Nix's cargo integration is even cooler since cargo is already quite deterministic, you just put in a SHA of the final folder cargo creates with all the dependencies.
This mostly works, but occasionally you need to override the generated derivations to add implicit dependencies that npm doesn't capture, or otherwise tweak packages that rely on quirks of npm.
Using npm doesn't demonstrate repeatability, but you could just as easily build a given application from fixed, verified source in this step.
It's absolutely essential that a package manager constrain the environment in which builds are performed. Without it, the packages are doomed to be built in a nondeterministic manner and potentially tampered with.
Not exactly the same as vendoring dependencies, but it's a decent compromise in some situations.
Habitat seems like an entirely new Chef product to enable what they're calling Application Automation.
"[...] automation must travel with the application, rather than be provided by the platform. Everything the application needs, from build dependencies, run-time dependencies, configuration, dynamic topologies, deployment strategies, secrets management, security auditing, and so on belongs with the application, because we're doing it for the application. Runtime and infrastructure layers exist to support the application, but the efficient building, deployment, and management of the application are decoupled from those layers."
Unfortunately, IMHO, the scope is a mile wide and an inch deep in terms of what chef brings to the table here. I can walk down that list and either think immediately of another tool that is dedicated to the specific task or the needs are so slight that a few ansible or bash scripts take care of the need. Sounds like they are aiming to occupy some space in the general "tooling & orchestration" arena, but it will be an uphill climb.
> Incredibly ambitious.
It is, but ambition is necessary if you want to build distributed production systems. BOSH does something similar in providing an all-or-nothing approach to configuring distributed systems built out of heterogenous software components spread across multiple machines.
Note here that the key is the heterogenous part -- if you have an undifferentiated flock of software, you can rely on a different abstraction to deploy them (anything from PaaS to a hand-rolled thing, though I don't recommend hand-rolling this stuff). But very few distributed systems are designed to rely on emergent behaviour. In fact that's usually what we're trying to avoid.
Where they appear to differ is that BOSH is quite strict about lifecycle -- you can't "truly" update configurations on-the-fly. You update the relevant YAML and then ask BOSH to update the world for you. It seems like a hassle, but if you couple it with git and Concourse pipelines, you get a complete, externalised history of every system configuration change you've ever made.
We already know that this top-down model of heterogenous distributed systems configuration management works better than trying to extend it from the machine up (as Cfengine and its various successors tried to do). BOSH has been deploying large systems for years and years. It was originally designed to deploy and upgrade Cloud Foundry without downtime, something it does with aplomb.
What I really like about Habitat over BOSH is the way it exposes the available knobs and dials. That kind of interactive discoverability is super helpful sometimes.
I was, however, struck by this claim:
> This architecture allows us to avoid the production cliff problem: the closer you get to production, the more difficult it is.
This is literally the problem that PaaSes solve for you. I've seen companies without PaaSes, with home-rolled PaaSes and with Cloud Foundry. In the latter case, the biggest obstacle is usually getting VPN access. And there are other PaaSes apart from CF. Habitat seems to be doing a kinda-sorta PaaS here (note the "Supervisor"), but without most of the feature plumbing that makes PaaSes operationally useful: routing, logging, service injection and the like.
Disclaimer: I work for Pivotal, we do the majority of the engineering on both BOSH and Cloud Foundry.
Rust gives us 3 things we needed, and it says so right on the website: fast, no segfaults, thread safety. Additionally, it's community is top notch. We've found the language to be great to work with, especially when you're refactoring. The type system makes it clear where it's possible for the code to fail, making it easier for us to reason about keeping the supervisor safe.
I wrote this while I was evaluating languages for the early habitat prototypes: https://medium.com/@adamhjk/rust-and-go-e18d511fbd95#.d35imt...
Like most languages idiom is learned while working with the language. Something I've found for instance is to try and use Iterators to get around sharing rules as they can encapsulate sharing patterns in a lot of cases.
The specific modules you mentioned are for multi-thread support, those are only necessary for sharing memory between threads.
Anyway, I think this is off-topic for this thread.
"build automation, IT automation, deployment automation, etc" would have saved me some confusion. Or simply compare yourself to other solutions out there will help me understand what category this new piece of utility and name fall into.
For a project that's smaller in scope but addresses the part about packaging the automation with the application, Joyent's ContainerPilot  looks interesting.
* Flexible versioning and configuration rollout
* Ability to deploy anywhere
* Ease of management
Concretely, they're touting their gossip-based configuration and topology management layer as a way to send configuration updates to service groups, and then letting the service groups apply the changes to all the group members. Contrast this with a more traditional way of propagating changes across a network by enumerating every single host/VM/container. I'd imagine another large component of the application automation would come in the form of tooling allowing the ability to deploy anywhere (bare metal vs. VM vs. container vs. IaaS/PaaS providers), but haven't seen that just yet. Maybe it's being covered in the live announcement.
I am always amazed at people's capability to give an extremely complicated answer to an extremely simple question. "What is being automated here?" "Software builds."
The docs/articles say habitat apps can run in containers, but why bother if your app can already be run in a container that achieves the same thing with even greater isolation (cgroups). Why put a self contained app in a self contained container application?
I'm not trying to put down Habitat, I'm trying to understand its role. As I currently see it, why would I choose this over the more mature Docker environment with a massive established ecosystem and user base?
Again I'm just trying to understand what I'm missing, if I'm missing something please point me in the right direction.
Im curious about how it works under the hood, it seems to solve 'all the problems' ;)
My skepticism comes from some of chefs past product releases, which have seemed rather, half baked.
Products like chef metal (provisioning), pushy (push jobs) or
This is in no way negative as I am a long time chef user and fan of chef.
If I need to pre-install all these services on my hosts before any of this will work, then how is that really different from uniformly deploying all my applications on an application server such as WebSphere? An EAR too contains all my configuration data, my bindings, my security settings etc. So...if you uniformly run WebSphere everywhere in your enterprise, it sort of accomplishers the same thing.
HOWEVER, if the habitat configuration is capable of talking to my F5 load balancer to create a resource pool, self register my app with my corporate DNS, create a DB schema and populate it with my tables, talk to my networking equipment and open all relevant firewall ports and do so when the application is deployed to a new environment, as well as undo all these actions if it's ever moved elsewhere, then you have piqued my interest...
Is it just me or are there a ton of other folks doing this? The idea of introducing yet another layer of stuff is scary, and there's lots of work left to do on Chef. In a lot of ways, they feel just like Nomad (coincidentally also on the front page)... I wish they would spend some time on their core products versus expanding into an overfilled market.
However it shows that Chef (Company) is still able to design, build and ship a new product, as an organization. That's a very positive sign. Until recently I thought when comparing Chef with Hashicorp: Hashicorp is so much faster in building and releasing usable stuff but they seem to feel the pain in growing too fast and too broad in terms of products.
So, in the end: It's a great thing, just also don't forget to put more love into Chef (software) and don't dissipate developers/time/budget :)
This thing uses kubernetes or this thing uses mesos
This other thing uses docker/rkt/containerd/lxd
This is the set of things we wrote in rust, etc, etc.
Bash is plenty good.
The problem is that everyone disagrees on what a tolerable real programming language is. Some refuse to use Python; others Ruby; others Scheme; others Lisp.
At least every can tolerate shell more or less.