In one of my ex-companies I worked with, we had a newbie coder who was just getting fascinated by Kubernetes. Unfortunately, every project this person touched would be left half-way by him and he'd move on to another project.
This company invested around $1 million into a very new Saas product that had a lot of potential. Now, I'm an old school guy. I like systems where I don't touch devops. The cost for such Paas is usually high, but it saves a lot of my time, which I value more than money (and it is valuable, in many cases). I advised the management to use something like AppEngine which is freakin' awesome for such kind of large scale projects.
Unfortunately, this newbie coder was management's pet. They bit the bullet and went on with his advice to use Kubernetes instead. The system randomly failed, they spent tons of time doing devops and microservices on what should have been a simplified monolith. The development time elongated to almost 2 years. By this time, there were competing offerings in the market for much cheaper.
This costed an entire team's morale, which lead to missed deadlines and product launches. This lead to many missed opportunities. For instance, there was a HUGE client we were demo'ing this Saas product. It failed spectacularly on a relatively light load (I've hosted Rails apps on AppEngine that can handle FAR better) and we lost that product and the client.
It was still not too late and I insisted on management identifying the root cause to switch to a managed Paas stack given how resource constrained we were. But this newbie coder turned it into a political situation ('mine vs yours'). I tried various ways to resolve this, but it didn't work out.
As a result, I said 'fuck this, I'm out of here' and I quit. In less than 3 months of my departure, about 10 people quit including one of the senior managers. In about 6 months, the company shrunk from a double digit to single digit company. The company almost went into bankruptcy.
The company lost its $1 million investment. Everyone left. The product development was put on indefinite hold. Finally, this newbie coder left as well. The founders had to rebuild their companies from scratch.
Co-incidentally, I went on to become an AppEngine consultant. I'm able to run unbelievably large monoliths in production, with almost zero downtime for 3 years in a row simply because I chose to avoid devops, and most importantly infrastructure complexity. I pay more, but it easily is worth my ROI. Most of the time, microservices aren't required and we can get away with a simplified monolith. And if you can get away with monoliths, you probably don't need Kubernetes.
> if you can get away with monoliths, you probably don't need Kubernetes
Right, especially if you're doing microservices the "right way" which means, each service has not only its own app setup but its own data store. I am responsible for (among other things) maintaining and scaling my company's infrastructure, and while we do have some vague concept of microservices (app, api (monolith), queuing system, various ML apis, headless browser, etc) it is always always ALWAYS "what's the simplest way to do this?" Microservices are a ton of moving parts, and adding in their own data stores is another can of worms.
We have a need for a scheduler. We are using Nomad internally. I chose this over Kubernetes. Why? I know that Kubernetes has more muscle behind it, can do more "things" we might need in the future, etc. I chose it because of its operational simplicity. It strikes a perfect balance of "scale these three things up" without needing a dedicated ops team to run it.
So far it has performed great, and I really think it comes down to limiting the number of moving pieces. Don't get me wrong, microservices are great if you have an ops team and a 50+ dev team all working on different things. But I often see microservices being pushed without warning people of the inherent costs involved.
I'm sure kubernetes is great, and I'm continuously re-evaluating where we are and what we need, but for now simpler is always better. If we do go with Kube, it will likely be a managed version so I don't have to touch the internals...not because I don't want to learn them (I definitely would want to know all the grimy details) but because I just don't have the fucking time to deal with it.
I think programming as a profession provides three main services:
1. Implementing user/client features (meeting demand)
2. Designing and maintaining an architecture flexible enough to continue adding features at a reasonable pace
3. Reducing complexity, which primarily involves picking the right primitives and reducing the number of variables as much as possible
2 and 3 are really just in service of 1, but the fact is that if you don't invest in flexibility/leverage/optionality and proactively keep yourself from getting swamped by an overwhelming number of "moving parts," your productivity will creep to a halt and formerly cost-effective features will no longer be economical, destroying your capability to provide customer value.
How do you use Nomad? Do you run Docker containers with it? I don't need the scaling, I need the part where new versions of code are autodeployed automatically, services are connected together, and basically things update without having to run ops checklists every time.
Kubernetes feels too heavyweight and deploying machines feels too snowflakey...
We use nomad a few different ways. One is system jobs which run DNS and Fabio (the networking fabric layer) on the host machines. Then, yeah, for all the apps and services we use Docker containers which I've found runs pretty well...with one caveat: make sure if you're spinning jobs up/down a lot you get a fast disk. For instance, we were running on 64gb ebs gpio volumes and Docker was starting to grind to a halt on them (we spin workers for our queue up/down quite a lot). We started using instances with ephemeral SSDs instead which has a bit more operational complexity on init, but overall works really nicely.
We also use Nomad for deployments as well. I wrote some deploy scripts by hand that create the Docker containers and load the templated Nomad jobs ie
I'd say as far as "take this container and run N instances of it and load balance requests at this hostname to them" Nomad has been pretty great. There is definitely some work involved getting the comm fabric all set up exactly how you want (Fabio does make this easier but it's still work). Consul now has Connect (https://www.consul.io/docs/connect/index.html) which I haven't looked at yet which might alleviate a lot of this. I think some of our complexity also came fromt he fact that we do have TCP services we needed to load balance and most fo the service fabric stuff forces HTTP onto everything.
Overall my experience with Nomad has been great. It's capable and really not too difficult to operate for one person who also has tons of other stuff going as well =].
This is super useful, thank you! Do you think all of this is worth doing, now that you can basically get managed Kubernetes for free from providers as long as you use their machines?
It feels like this is much easier than rolling your own Kube, but not easier than using the managed version...
Not sure, honestly. I've never used Kube, just taken a preliminary look at the docs and been scared away by how much abtraction there is. While providers may manage it for us, I'm not sure to what extent they manage it. We're on AWS and I haven't been super happy with the responses/response times of their support, so when dealing with unknowns I'd rather not rely on someone else.
That said, Nomad hasn't been without problems. It's just that the problems seem to be easier for one person to solve. I set all this up almost a year and a half ago and haven't touched it much since, so it's possible both Nomad, Kube, and and managed services have come a long way and now is a good time to re-evaluate.
I can't see how Kubernetes belongs to this problem. It's just the tool for getting specific things done, not some kind of silver bullet. You can build your application layer around Kubernetes, leveraging its primitives and integrating your services tightly, or you can leave it as be (wow).
I currently maintain few clusters with managed control plane for hosting some monolith apps. Just a couple of fat worker nodes for failover, ingress with nice metrics and benefits of containerization... and that's it. No need to push for fully decoupled microservice architecture with service mesh that eats more CPU than your app (sarcasm, but not so far from the reality). Just use your tool right.
This company invested around $1 million into a very new Saas product that had a lot of potential. Now, I'm an old school guy. I like systems where I don't touch devops. The cost for such Paas is usually high, but it saves a lot of my time, which I value more than money (and it is valuable, in many cases). I advised the management to use something like AppEngine which is freakin' awesome for such kind of large scale projects.
Unfortunately, this newbie coder was management's pet. They bit the bullet and went on with his advice to use Kubernetes instead. The system randomly failed, they spent tons of time doing devops and microservices on what should have been a simplified monolith. The development time elongated to almost 2 years. By this time, there were competing offerings in the market for much cheaper.
This costed an entire team's morale, which lead to missed deadlines and product launches. This lead to many missed opportunities. For instance, there was a HUGE client we were demo'ing this Saas product. It failed spectacularly on a relatively light load (I've hosted Rails apps on AppEngine that can handle FAR better) and we lost that product and the client.
It was still not too late and I insisted on management identifying the root cause to switch to a managed Paas stack given how resource constrained we were. But this newbie coder turned it into a political situation ('mine vs yours'). I tried various ways to resolve this, but it didn't work out.
As a result, I said 'fuck this, I'm out of here' and I quit. In less than 3 months of my departure, about 10 people quit including one of the senior managers. In about 6 months, the company shrunk from a double digit to single digit company. The company almost went into bankruptcy.
The company lost its $1 million investment. Everyone left. The product development was put on indefinite hold. Finally, this newbie coder left as well. The founders had to rebuild their companies from scratch.
Co-incidentally, I went on to become an AppEngine consultant. I'm able to run unbelievably large monoliths in production, with almost zero downtime for 3 years in a row simply because I chose to avoid devops, and most importantly infrastructure complexity. I pay more, but it easily is worth my ROI. Most of the time, microservices aren't required and we can get away with a simplified monolith. And if you can get away with monoliths, you probably don't need Kubernetes.