
Ask HN: Nomad (Hashicorp) vs. Kubernetes? - albertlie
Hi all,<p>I&#x27;m currently looking for container orchestration tools in my current company. We have our applications across multiple data centers (GCP, AWS, Alicloud).<p>From my research, I found out that the most common ones are Nomad and Kubernetes. Anyone has experience running either Nomad or Kubernetes in production? Furthermore, maybe the drawback and benefit from your experiences?<p>Thanks
======
SkyRocknRoll
We use nomad in production for last 3 years. Recently we thought of moving to
kubernetes and did a prototype for one of our products because of community is
big and growing. After looking at the sheer complexity we decided to stick
with nomad. The best part about nomad is it solves single problem very well
and it is very simple to understand and maintain. Developers are able to grok
the nomad in matter of minutes which is a big win for us because they are the
one who writes nomad files.

------
djhaskin987
Kubernetes and Nomad are resource schedulers. Resource scheduling is a hard
problem. For example, Linux is also a resource scheduler.

Right now, we're seeing something similar to the OS wars in the 80's, 90's and
2000's. Back then there was Solaris, Linux, Unix, BSD, etc. In the 2000's,
Linux won. After that, the sheer force of community made it easy to deploy and
use.

The community around Linux mitigated its complexity by making distributions --
opinionated choices around how to set up linux, including what package manager
to use, what init process to use, etc.

Kubernetes is, in my opinion, far more complicated than Nomad, but it's also
more feature rich. To mitigate that complexity, quasi-distributions of
kubernetes are emerging, like kops (
[https://github.com/kubernetes/kops](https://github.com/kubernetes/kops) ) and
kubespray ( [https://github.com/kubernetes-
incubator/kubespray](https://github.com/kubernetes-incubator/kubespray) ). My
personal favorite is Typhoon (
[https://typhoon.psdn.io/](https://typhoon.psdn.io/) ), because it uses a
"package manager" (software delivery mechanism) that allows you to declare
dependencies between packages and also allows you to deploy the entire cluster
from the ground up -- Hashicorp's Terraform (
[https://www.terraform.io/](https://www.terraform.io/) ).

The kubernetes package manager is helm ( [https://helm.sh/](https://helm.sh/)
), but if I were you I'd wrap helm charts in terraform modules using the helm
provider ( [https://github.com/mcuadros/terraform-provider-
helm](https://github.com/mcuadros/terraform-provider-helm) ).

By contrast, Nomad is much simpler to set up and has a much better modular
design. But then, so did the OS GNU Hurd. And no one uses that anymore.
Kubernetes is much more monolithic in design (huh, so is Linux).

Therefore, I would choose kubernetes, but treat it like an OS. It's
complicated. Don't choose what ingress you will use. Instead choose what
_distribution_ you will use, and let the distribution guide your decisions,
unless you have a good reason.

Realize that you're learning an OS (resource scheduler = os in my head).
Kubernetes has more complexity than Nomad up front, but at least it's not
accidental complexity. K8s calls the resource scheduler problem what it is --
hard, and treats it appropriately.

The other safe bet is Terraform as the package manager. Choose a kubernetes
distro that uses it (Typhoon, Kops). It is getting _huge_.

Kubernetes' extensible API is what really makes it cool. Having a prometheus
operator ( [https://github.com/coreos/prometheus-
operator](https://github.com/coreos/prometheus-operator) ) automatically
monitor your services for you is pretty nice.

Finally, don't expect the stability you see in the Linux OS. Kubernetes is
still coming out of the primordial ooze, though it's way more mature than
Nomad in its community. Still, when a google search for how to do something on
a cluster mostly turns up github issues, you know we're still in the early
stages.

That said, Nomad is way easier to set up, and supports things like raw
execution, JVM direct execution, and chrooted exec, so if you can't use
containers at least you can still use a cluster. You'll be rigging everything
yourself, but raw execution actually gives you a lot of rope (
[https://github.com/hashicorp/nomad/issues/150](https://github.com/hashicorp/nomad/issues/150)
). So if you need to do something really custom, Nomad might not be so bad.
Kind of like embedded Linux.

~~~
gtirloni
Could you comment a bit more on how Nomad has better modular design? Almost
everything in Kubernetes is replaceable so what makes Nomad better in that
regard? Thank you

~~~
djhaskin987
Sure. What I mean by that is that if you use kubernetes, you will be using
kubernetes service Discovery, endpoints, ingress, and volumes, where nomad is
_strictly_ the scheduler. For example you can easily set up nomad with consul
service discovery and then use the same consul instance for service discovery
of services living outside nomad, such as legacy applications. It's hard to do
something like that with Kube services because kubernetes comes with its own
service discovery mechanism.

For more comparison between the two schedulers coming from the point of view
of someone who chose Nomad, look at this article
[https://medium.com/@copyconstruct/schedulers-kubernetes-
and-...](https://medium.com/@copyconstruct/schedulers-kubernetes-and-
nomad-b0f2e14a896) . I especially found his idea of incremental infrastructure
improvement interesting, near the bottom if of the article.

~~~
djhaskin987
I just wanted to apologize, I incorrectly refer to the author of the article
above as "him". It's a "her", Cindy Sridharan (
[https://twitter.com/copyconstruct](https://twitter.com/copyconstruct) ).

------
jeskie
Have you looked at Mesosphere DC/OS?

