37signals is not like the typical large-scale startup. They have an extremely small team (around 30 people?), and just a couple of products.
Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams.
When you've got a couple products that have been in maintenance mode for 10+ years and then 2 just two primary products, both of which are on the same stack, and you can predict your workloads way out into the future (because you charge money for your services, don't do viral go-to-markets, and don't have public services), there simply isn't much of a win to dynamic scheduling. You can, in fact, just have a yaml file somewhere with all your hosts in it, and write some shell-grade tooling to roll new versions of your apps out.
A lot of the reflexive pushback to not using k8s seemed like it came from people that either didn't understand that 37signals was doing something closer to static scheduling, or that don't understand that most of what makes k8s complicated is dynamic scheduling.
Most startups that I see trying to go with microservices too early do so while keeping a shared database between the services, so they're not really microservices, but a distributed monolith. This turns into a massive, massive pain.
Doing microservices well means building out templates for CI/CD pipelines, templates for observability, figuring out how best to share (or not share) common credentials like third-party API keys, setting up a service mesh, setting up services like Backstage and Buf... which inevitably requires hiring dedicated engineers at enormous cost.
If you can set up a monolith, why would you switch to microservices? So that you can have a smaller container image and faster autoscaling? So the developers on the second and third teams you hire can wait a little less time for CI tests to run?
It's a pretty big mistake to adopt microservices too early.
Sure, but I'm not thinking of small startups, but rather large ones like Doordash, Airbnb, Box, Square, Stripe; these places legitimately have lots of services, complex coupling patterns, and a need for some kind of indirection and dynamic placement to enable teams to get things done without coordinating every single PR.
37signals presumably doesn't have any of those problems. They should be using K8s. In the previous thread, I suggested that I'd consider Nomad if I were them, but really, they don't need Nomad either. They need a yaml file mapping containers to hosts, and some tooling to do smooth deploys and to hook up a relatively static service discovery map. Which: that's what they built.
Nitpick: it may be helpful to not use the term "startups" when you mean companies like Doordash, Airbnb, Stripe etc. Those companies _were_ startups... a very long time ago. Same with 37signals.
This advice offered in the hope that it may improve communication!
You are talking about a handful of unicorns, not just large startups, and even then only a subset of them — Shopify with their widely successful monolith comes to mind. By definition their requirements in their current unicorn state can't be the default.
Yet we collectively behave as they are and everyone needs dynamic scheduling (and the rest of the kitchen sink).
> Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams.
This is the crux. It's Conway's Law in action. The promise of Kubernetes to a large org is that you can split the baremetal and OS layer into one team that manages everything up to Kubernetes, and then that's the common interface where all other teams deploy their applications into. And besides the team separation, the value is that you have a central layer to put automatic policy enforcement, engineering doctrines, and so forth.
while i agree with you that kubernetes allows you to split the stack down to individual "specialisations", i don't see how this is different with kubernetes compared to "traditional" Virtual machines?
Usually the differentiator is either the container or VM. Everything beneath that gets managed by some kind of operations team (or teams if you are really large or specialized), while the stuff running ontop of the infrastructure is usually done by developers/application maintainers.
> what makes k8s complicated is dynamic scheduling.
… Which almost no startup or otherwise ever will need. Creating complex stuff for workloads you will never ever have. You hope to have them, but that’s called premature optimisation. And then you still mostly likely fall in the bracket of a company that will never need it.
I work for a company that routinely deploy very large scale software to airlines/airports/rail companies around the world. Millions of lines of mission critical server and mobile/browser/desktop client code.
We do it without the cloud, without micro-services, without Kubernetes etc. Just straight forward good old fashioned client/server monoliths. It’s simple. It works.
The reality is that 99% of people who think they need Kubernetes don’t actually need it. Almost all problems in software development are caused by the developers themselves. Not by the actual business problems.
Who said anything about a large-scale startup? Kubernetes is approachable all the way down to N=1 employees.
I strongly disagree with your take on static vs dynamic scheduling. Static scheduling ties your hands early. In a mature organization, it is very much an optimization.
Dynamic scheduling forces a cattle-not-pets mentality out of the gate, which is great. It also gives you all the knobs to figure out HA, right-sizing and performance that you'd ever want, for whenever you're ready for them. It's considerably more laborious to rearrange or tune things with static scheduling. I've run the gamut here and Kubernetes is by far the easiest and most approachable way to manage a fleet from 1 to $BIG that I've ever encountered. If you want to treat it like a static scheduler, that is also trivial. It's not like there's some huge cost to doing so. It's basically a NOP.
37signals blew off their foot by doubling down on their sunk costs (read: capex) of metal. They clearly don't want to not think about KVM and F5s and SSH keys and all the other odds and ends that entirely solved away by managed services for reasonable prices.
Which is it? Are they too big for the cloud or too small? You can't have it both ways.
You're likely looking at the problem from a different angle. Kubernetes is an ecosystem for managing software deployed on a fleet. There's value in that even if it's two or three hosts. What size of an org is too small for Terraform? Puppet? CI/CD? Docker? Some bash scripts?
All these tools should fundamentally solve some piece of the puzzle for you. Kubernetes on a managed platform just happens to cover many pieces that you still need to solve otherwise. It may be more novel than some other technology but it's not less proven or difficult, fundamentally.
Most of the criticism I see here is literally just FUD. I run Kubernetes at home. It's fine. It's less work than anything comparable I've ever encountered.
It's not a huge cost. This is ludicrous. Sign up for a free account on GCP and walk through their getting started guide and you'll have stuff running in under an hour.
> your "fine" rests on a base level of knowledge which is enormous, certainly incompatible with p99 of users
I doubt it. I have far, far more experience running services on bare metal pizza boxes in 48U racks in PoPs all over the world, but that is a base level of knowledge that actually is enormous and certainly incompatible with p99 of my pool of potential colleagues.
If you can figure out how to run a serverless app, you can figure out Kubernetes. It's not rocket science.
This is the attitude I recoil from. Some of the technologies you listed --- CI is an example --- earn their keep so well that they're essentially universal best practices (at least for serverside software). But others don't, and you can't use a list that includes, like, "source control" as a way to launder in the idea that Kubernetes is a universal best practice. It is not. It earns its keep in some places and not in others.
The only best practice I strongly preach is using source control to define your infrastructure all the way down to the versions of binaries serving your customers. Kubernetes is a means to an end there, when used in combination with "gitops". It's very compelling and a very malleable pattern.
As far as I can tell the other alternatives are to either go low and own the kernel on up, which means a lot more depth in your stack. Albeit this is the traditional systems curriculum, it still requires a lot of expertise that is relatively uncommon. I was once like you, I suspect, and our rose-colored glasses for the days of yore make going low more appetizing than I think is reasonable from a business perspective. No one at a startup should ever care about iptables.
The other end of the spectrum is stapling your software to managed services that range from portable (S3) to not (GCP's PubSub). For most startups I'd reckon this is actually the preferred approach as portability can be solved after you find some sort of product market fit. My reservation here is that going this route often blinds folks from understanding how to solve these problems yourself, without the handholding of a big cloud.
Here is where Kubernetes shines. It's the best of both worlds, and the lowest common denominator is relatively portable but at a high enough level that you can be productive. Surely our experiences here differ but mine thus far has been nothing short of positive in the last couple of years. Before then? Mostly a shitshow without deep expertise, but today you can get by without learning much at all.
In comparison to Kubernetes, the closest thing I've used that wasn't kubernetes was a bunch of home-rolled Terraform and puppet and shell scripts. Not too different than what 37signals is bragging about. But it sucked. It's fragile and complex and if the one person who wrote 95% of it leaves your company you are hosed. Kubernetes, when used in an appropriate manner, unshackles you from this sort of thing. Sure, YAML is terrible but at least your YAML looks a lot like my YAML or that YAML on stack overflow.
Kubernetes is fundamentally the next easiest thing behind whatever the equivalent to Heroku/Cloudrun/Fly.io.
If you think it's easier to run a service on a VPS by scp'ing a tarball and ./'ing the server then you're not being honest with yourself about your sysadmin hobby/addiction affecting your work. If you're skilled and competent, good for you! I am too, and we could run out servers happily together forever. But it's sadly a lift to ask newcomers to follow along. The industry is moving away from this level of depth as far as most developers are concerned.
Large-scale startups use dynamic-scheduled cloud services in part to reduce coupling between teams. Every service --- and there are dozens --- is scheduled independently, and new teams can get spun up to roll out new services without too much intervention from other teams.
When you've got a couple products that have been in maintenance mode for 10+ years and then 2 just two primary products, both of which are on the same stack, and you can predict your workloads way out into the future (because you charge money for your services, don't do viral go-to-markets, and don't have public services), there simply isn't much of a win to dynamic scheduling. You can, in fact, just have a yaml file somewhere with all your hosts in it, and write some shell-grade tooling to roll new versions of your apps out.
A lot of the reflexive pushback to not using k8s seemed like it came from people that either didn't understand that 37signals was doing something closer to static scheduling, or that don't understand that most of what makes k8s complicated is dynamic scheduling.