Ask HN: Nomad (Hashicorp) vs. Kubernetes?

[albertlie]

Hi all,

I'm currently looking for container orchestration tools in my current company. We have our applications across multiple data centers (GCP, AWS, Alicloud).

From my research, I found out that the most common ones are Nomad and Kubernetes. Anyone has experience running either Nomad or Kubernetes in production? Furthermore, maybe the drawback and benefit from your experiences?

Thanks

[SkyRocknRoll]

We use nomad in production for last 3 years. Recently we thought of moving to kubernetes and did a prototype for one of our products because of community is big and growing. After looking at the sheer complexity we decided to stick with nomad. The best part about nomad is it solves single problem very well and it is very simple to understand and maintain. Developers are able to grok the nomad in matter of minutes which is a big win for us because they are the one who writes nomad files.

[djha-skin]

Kubernetes and Nomad are resource schedulers. Resource scheduling is a hard problem. For example, Linux is also a resource scheduler.

Right now, we're seeing something similar to the OS wars in the 80's, 90's and 2000's. Back then there was Solaris, Linux, Unix, BSD, etc. In the 2000's, Linux won. After that, the sheer force of community made it easy to deploy and use.

The community around Linux mitigated its complexity by making distributions -- opinionated choices around how to set up linux, including what package manager to use, what init process to use, etc.

Kubernetes is, in my opinion, far more complicated than Nomad, but it's also more feature rich. To mitigate that complexity, quasi-distributions of kubernetes are emerging, like kops ( https://github.com/kubernetes/kops ) and kubespray ( https://github.com/kubernetes-incubator/kubespray ). My personal favorite is Typhoon ( https://typhoon.psdn.io/ ), because it uses a "package manager" (software delivery mechanism) that allows you to declare dependencies between packages and also allows you to deploy the entire cluster from the ground up -- Hashicorp's Terraform ( https://www.terraform.io/ ).

The kubernetes package manager is helm ( https://helm.sh/ ), but if I were you I'd wrap helm charts in terraform modules using the helm provider ( https://github.com/mcuadros/terraform-provider-helm ).

By contrast, Nomad is much simpler to set up and has a much better modular design. But then, so did the OS GNU Hurd. And no one uses that anymore. Kubernetes is much more monolithic in design (huh, so is Linux).

Therefore, I would choose kubernetes, but treat it like an OS. It's complicated. Don't choose what ingress you will use. Instead choose what distribution you will use, and let the distribution guide your decisions, unless you have a good reason.

Realize that you're learning an OS (resource scheduler = os in my head). Kubernetes has more complexity than Nomad up front, but at least it's not accidental complexity. K8s calls the resource scheduler problem what it is -- hard, and treats it appropriately.

The other safe bet is Terraform as the package manager. Choose a kubernetes distro that uses it (Typhoon, Kops). It is getting huge.

Kubernetes' extensible API is what really makes it cool. Having a prometheus operator ( https://github.com/coreos/prometheus-operator ) automatically monitor your services for you is pretty nice.

Finally, don't expect the stability you see in the Linux OS. Kubernetes is still coming out of the primordial ooze, though it's way more mature than Nomad in its community. Still, when a google search for how to do something on a cluster mostly turns up github issues, you know we're still in the early stages.

That said, Nomad is way easier to set up, and supports things like raw execution, JVM direct execution, and chrooted exec, so if you can't use containers at least you can still use a cluster. You'll be rigging everything yourself, but raw execution actually gives you a lot of rope ( https://github.com/hashicorp/nomad/issues/150 ). So if you need to do something really custom, Nomad might not be so bad. Kind of like embedded Linux.

[gtirloni → djha-skin]

Could you comment a bit more on how Nomad has better modular design? Almost everything in Kubernetes is replaceable so what makes Nomad better in that regard? Thank you

[djha-skin → gtirloni]

Sure. What I mean by that is that if you use kubernetes, you will be using kubernetes service Discovery, endpoints, ingress, and volumes, where nomad is strictly the scheduler. For example you can easily set up nomad with consul service discovery and then use the same consul instance for service discovery of services living outside nomad, such as legacy applications. It's hard to do something like that with Kube services because kubernetes comes with its own service discovery mechanism.

For more comparison between the two schedulers coming from the point of view of someone who chose Nomad, look at this article https://medium.com/@copyconstruct/schedulers-kubernetes-and-... . I especially found his idea of incremental infrastructure improvement interesting, near the bottom if of the article.

[djha-skin → djha-skin]

I just wanted to apologize, I incorrectly refer to the author of the article above as "him". It's a "her", Cindy Sridharan ( https://twitter.com/copyconstruct ).

[jeskie]

Have you looked at Mesosphere DC/OS?