Did K8s reinvent VMs though? People keep forgetting that vm!=container and the security posture is completely different. Also, for the amounts of complexity K8s adds it's not worth the overhead in 95% of cases (i get it that it's cool and everyone and their dog want to claim they have experience with it - I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered)
Kubernetes isn't that complicated, it's distributed computing that is complicated. Kubernetes makes it as simple as it can be without oversimplifying it.
Just wanted to chime in and hard agree on this. I remember the world where people were trying to build things like kubernetes before it existed. There was a period where enough people had left google post ipo to realize that they needed something like borg but nothing really existed like that outside of elgoog.
Twitter had Aurora, then elsewhere mesos and mesosphere popped up and it seemed like there were going to be about 100 different frameworks until Kubernetes dropped and basically ate the industry alive. K8s feels pretty stable from where I’m sitting.
Not to mention all the various in-house systems in the 2010-era. I worked at a small place back in 2013 that had built their own container platform about a year before Docker was released.
It was more like workqueue than borg. But it worked well enough. Eventually we got tired of maintaining our own snowflake scheduler and switched to Kubernetes.
But I'm also old enough to remember pre-borg systems used in HPC. Maui, TORQUE, etc. I wasn't surprised batch scheduling finally got used for service deployment.
Even in the post docker world people were trying to build container platforms on docker that resembled other things but had their own issues. I definitely had to maintain one of those for three years.
Kubernetes is kind of stable except for the relentless deprecation and introduction of features. I.e. ingress going from v1beta1 to v1 and dropping any support for the v1beta1 defs, service accounts no longer having tokens associated with it, etc.
It really makes for a constant churn where there's a good change something breaks if you update kubernetes. Either that or you pick a version and stick with it for 2 years while it's supported and then jump to the new one and fix all the breakage.
Running a microservices architecture on a shared cluster is complicated.
* Because you are running multiple workloads on the same kernel, you need to protect them from each other, from both correctness and performance perspectives.
* Because the workloads have different scaling characteristics, you need to solve a bin-packing problem to make efficient use of the resources.
* Because of said bin-packing, workloads move around a lot, so you invite a service discovery problem much more intense than classic DNS is meant to solve.
* Because normal software only knows how to use DNS, you invite the need for sidecars and virtual network overlays.
When you have this set of problems then k8s seems appropriate. I work at a company with this set of problems but no Kubernetes, and boy is it a production. But my takeaway from that is less "use Kubernetes" and more "try not to have those problems." Use relatively monolithic architectures, one or a few services on their own stable pools of nodes, with boring old reverse-proxy load balancers and DNS, for as long as you possibly can.
I have come at this problem from a bit of a different angle by asking what is the closest I can possibly get to the hypothetical dream state of everything is automated, autoscaling blah blah blah as possible with the absolute smallest budget in terms of not only actual costs but time budget as well.
I only know the GCP ecosystem kind of well so I don't fully know to what extent these things exist in AWS and Azure but there I think there is a really nice path you can get on with the serverless route that skips K8s entirely but keeps you very well aligned in case you ever need to "upgrade" or get out of the GCP ecosystem.
I write very stock standard gRPC services and then put them onto Cloud Run (which has a very Heroku like workflow) and stick https://cloud.google.com/api-gateway in front of things and now my API is running on the exact same setup as any other service Google is running in production. Huge amounts of logic get moved out of my code base as a result.
Now it's all declarative where I just say what I want to happen and I don't have to think about much else beyond that because it too is using that same internal GCP infrastructure to handle all the complicated parts around what to do when things go wrong.
But to me they are all extremely heavily aligned with the K8s path so the lock in certainly doesn't feel as scary.
You have to still ask the question why not just deploy the monolith to the VM and move on until you need to think about anything else?
For me it’s not JUST an investment in the future but in the very real immediate benefits it’s the huge amount of code I don’t have to write and all the insanely advanced stuff I get for free.
To give a quick example of each though. On the code side I don’t think about things like authN, authZ, retry logic, health checking, most security outside of things like input validation, logging, tracing etc. All of that now is just a configuration setting for me.
Then on the advanced features side, just to give one example. If I set a security policy saying this service can do the following actions on another service and I don’t end up using most of those permissions it will automatically notify me in the future to let me know and help me rewrite the security policy to only use the things I need in practice, it can even help me test that new policy to ensure things don’t break… That was previously basically an entire persons job to find and fix those kinds of things, usually manually because the amount of code needed to automate it is painful to think about. Now it’s just another feature I pick up for free.
K8s is trying to do distributed computing on a low-level OS platform that was never developed or designed for this purpose. Look at things like plan9 to see how much easier and more elegant it could be. Sure, plan9 doesn't provide k8s' autoscaling and provisioning features out of the box but the basic building blocks are all there. (And no, Linux is nowhere close to providing these patterns because custom user-space implementations of kernel-side features are only made possible on a totally ad-hoc basis.)
If i where you i would stop use argumentation's like that.
OpenVMS has still the best Clustering tech...why is "nobody" using it? And btw p9 is very much used (WSL for example).
Linux is for sure not the best kernel, not for server not for desktop and not for mobile...it's good enough and everyone circles around it, it has some bacon for everyone (even if the bacon is sometimes disgusting).
You have to admit that it's tuned towards stateless and loadbalanced workloads, though. Not every workload fits into the k8s box. I like k8s but for everything else, k8s can make things overly complicated.
I don't know if I would agree with that. Using Kubernetes on a good day is not complicated, but it does have a lot of moving parts. There are many different flavors and distributions of Kubernetes, a lot of concepts that can prevent your workloads from running (because system pods taking too much resources, affinity, persistent storage, kubelet/etcd/apiservers cyclic dependencies, x509 certificates). An actually HA Kubernetes deployment needs an external load-balancer for no reason at all (kubelet could talk to multiple apiservers like all "distributed system" software does, but that's still not implemented).
> Kubernetes makes it as simple as it can be without oversimplifying it.
What about something like Hashicorp Nomad and Docker Swarm? The popularity of the former and the maintenance status of the latter aside, I think they achieve much of the same as Kubernetes in ways that are simpler, which is enough for the majority of deployments out there.
For example, most pieces of software that run in containers have a docker-compose.yml file which can then be fed into Docker Compose to launch an environment on a single node, say for local testing and development, or to just explore a piece of software in a throwaway environment. What Docker Swarm does, is take basically the same specification and add the ability to run containers across multiple loads, do networking across them in a reasonably secure way, whilst being able to set up resource limitations etc. as needed, as well as scale the containers across the available nodes and manage storage with volumes, bind mounts or even plugins for something like GlusterFS or just NFS.
Docker Swarm doesn't concern itself with the concept of Pods because you don't always need those - regular containers can be enough without the additional abstraction in the middle. Docker Swarm doesn't concern itself with the concept of a Service, since you can just access containers based on their names through the built in DNS abstraction, especially if you don't need complicated network isolation (which you can also achieve at server level). Docker Swarm doesn't really care about an Ingress abstraction either, since you can just make your own Nginx/Caddy/Apache container and bind it to ports 80 and 443 on all of the nodes where you want to have your own ingress. No PersistentVolume and PersistentVolumeClaim abstractions either, since the aforementioned bind mounts and volumes, or network storage are usually enough. And the resource usage and API are exceedingly simple, you don't even need to worry about service labels or anything like that, since in most cases you'll only care about the service name to access the container through.
If I built my own container orchestrator, I'd strive for that simplicity. Seems like projects like CapRover also recognized that: https://caprover.com/ Same with Dokku: https://dokku.com/
If you're in a company that has never really run advanced A/B tests or doesn't really need to do complex 2-stage DB migrations or blue-green deployments, circuit breaking and other fancy stuff, there's not that much use in going with Kubernetes, unless you really just want to hire for it and also pay someone else to manage it for you.
Personally, with tools like Kompose https://kompose.io/ I'd advise that you start with Docker and Docker Compose at first (locally, or for dev environments) and then branch out to Docker Swarm or Nomad, before eventually migrating over to Kubernetes, if you need to, maybe with something like K3s or K0s clusters at first. Or maybe even Portainer/Rancher, since those make the learning curve of Kubernetes far more tolerable. Or, at the risk of increasing the complexity of your deployments, go with Helm as well because the templates for Deployments and other objects that Helm creates by default are surprisingly useful in avoiding YAML hell.
Of course, some say that Docker Swarm is dead and you shouldn't ever touch it, which is why I mention Nomad (even though HCL is a little bit odd at times), which is also great, with the added quality of supporting non-container deployments. Either way, ideally look for a way to run apps in containers because they feel like the "right" abstraction, whilst finding the best fit for the features that you actually need vs the complexity introduced.
In short: containers are pretty good, running them has lots of options, although some are indeed complicated. Kubernetes is not the simplest way to run them, in my experience.
No one (not enough people?) asked (cared?) for this about the JVM for like 25 years till Linux Cgroups came around and Docker capitalized on it. Its not VM v/s container question that we care about. (I am aware this is a very poor comparison but so is comparing a vm to a container, which is my point)
K8S brings complexity yes, and in a consistent, predictable way. The biggest difference between a k8s based shop vs a DIY VM/Container shop is the former can (in theory) hire some people with intermediate K8S experience, and have them sort out the mess that the people without that experience (or more typically the ones who don't have clear requirements) may have created. With k8s it becomes more predictable to find the boundaries and start pulling things apart in production and make incremental changes. Without k8s, making changes to a production setup is a nightmare that folks who have done that sort of thing don't ever want to repeat.
> So your point is that once you learn how to use the tool it's easier to use the tool?
Kubernetes is not just "a tool", it's a platform. So if you know the k8s fundamentals, being dropped in an unfamiliar environment should not be a problem. It's the same as knowing your way around a Linux system, you might not be familiar with that specific distribution you're thrown into, but you should be able to explore the system and figure stuff out.
> Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
Any engineer who claims to know a bit more than the basics of k8s? It being a software, "devops", SRE, "platform", ... engineer doesn't really matter, I've been labeled all of the above.
I'll never claim k8s is perfect, I have plenty of issues with it, but very few of them are an architectural problem, and I've yet to encounter a platform that attempts to address all the issues k8s does. But you seem to be very critical of it, so what would you propose as a viable alternative?
We do (using Cluster API) and yes we keep it now within 2 releases from latest. Most of the work required to keep it up to date is not on k8s side tho…
Edit: I can answer questions on how we do it or maybe work on blog post if you’re seriously asking
I am asking because I've seen the work that goes into keeping a control plane (for anything other than toy applications) running. It's non-trivial and requires a lot of know-how about many not well documented k8s features. You may have figured a better way of doing it or maybe you have reduced complexity by having a more opioninated setup.
How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services) Are you running your workloads in the cloud or on dedicated bare metal machines?
I’d love to hear more details/chat about what aspects you found most difficult specifically with running control plane (not on gke/eks/aks). I’m looking at how I could solve this more generally for businesses ;)
> How many nodes (ballpark) are in your cluster, how big is the team supporting it and how many developers use the cluster you've stood up? (For how many services)
Up to upper hundreds of nodes/tens of thousands of cores (beefy nodes) per cluster in biggest ones (several geo/cloud-distributed clusters). Tbh what’s giving me more grief is number and diversity (different clouds/different setups) of clusters not individual cluster sizes. Original team that built/proved the design was only 3-4 devs but k8s was only part of what we did so it’s more like 1-2 if full time. Number of engineers that deploy there is probably close to 50 now with at least twice as many services (not all actively developed or in-house).
> Are you running your workloads in the cloud or on dedicated bare metal machines?
GCP and AWS VMs. We managed to stay away from anything managed except block and object storage so baremetal is in store for us just wasn’t the time yet. One of my main goals developing our current design was to keep this door open when the time comes and the setup will be largely the same as in cloud (with a few additions like 3rd party CSI for ISCI/NVMeOf storage)
Problems stemmed across multiple dimensions: limits on the numbers of nodes in the cluster (had to split into 2 clusters - think thousands of nodes), crds, weird network setup issues, daemon sets for storage/logs, persistent volumes, rough edges of the tooling operating the cluster (people tell you that with k8s you write less scripts! But in reality depending on your cluster size the scripting moves to the admin side.
This does not include actually upgrading k8s which was a nightmare and having to keep the underlying nodes patched.
Did you run into ipv4 subnet limitations wrt to node count or something else? 5k is max supported node count for a while now but running it with a tunnel-based cni (as is typical in cloud) would probably be too wasteful so it’s likely only an option for when you can configure l3 on your own switches. Or maybe using ENIs in ec2. I found that ipv4 space requires very careful planning especially if you plan on peering clusters together.
> weird network setup issues, daemon sets for storage/logs, persistent volumes, rough edges of the tooling operating the cluster (people tell you that with k8s you write less scripts! But in reality depending on your cluster size the scripting moves to the admin side.
Network with Cilium has been mostly “setup once and forget”. But yeah all the addons will require constant upkeep in a self-managed scenario. As far as configuration management goes I agree it’s a big hole which we had to fill with our own DSL configuration framework (think Helm3 but with Starlark). Now there are products like Pulumi which can offer similar API
I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubernetes isn't that much more complicated either.
A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.
Service Fabric needs to die. It is an abomination on software.
Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in.
No thanks.
Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in.
> Windows doesn’t have “distributions”, so there’s less need for containers in that space.
I don't follow. What does distributions have to do with need for containers?
> What does distributions have to do with need for containers?
A lot (but not everything).
It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc...
In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters.
Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself.
Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable.
[1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is.
[2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc
Even if Windows containers are not as portable, the ability to manage them from the same Kubernetes controlplane that manages Linux containers seems pretty powerful. It also enables you to do it outside of Azure/Service Fabric on other clouds or metal.
You're trading one convenience for a world of hurt if you use Kubernetes with Windows. Just take a casual glance at the "Windows containers" issues list on GitHub: https://github.com/microsoft/Windows-Containers/issues
Some of the very recently closed issues were jaw-dropping, such as totally broken networking in common scenarios.
DNS resolution is very different in Windows compared to Linux, making much of the "neatness" of Kubernetes pod-to-pod communication not work.
There is no maximum memory limit in Windows kernel "job objects" (equivalent to cgroups), so one memory leak in one pod can kill an entire node or cluster. This is very hard to solve, and I've seen it take out Service Fabric clusters also.
Thanks for sharing, it all seems terrifying but still very interesting to explore. I was curious how the Windows nodes would even join - turns out they support cloud-init just like Linux machines.
Not so much, no. Linux is the kernel, and all Linux distros, every single one, employs Linux as its kernel, or else it would not be Linux. Stepping back from my pedantry, I believe what you must mean is that different Linux distributions will customize the kernel a bit. That doesn't change what it is. Just because you have automatic windows and I have cranks, and you have leather interior and I have plush doesn't mean our vehicles aren't the exact same year, make and model.
Sure, I can imagine, but that is merely a bug, or perhaps the compatibility issue stems from misuse of a general but inaccurate claim of "Linux compatible." My expectation is specific compatibility is stated plainly in the documentation and on the box, if there is one, without ambiguity.
Note that because of Linus's strict policies on userspace compatibility, A>B is basically guaranteed to always be safe, and only A<B will ever cause problems, unless distros add patches that break things.
I e heard this before, but the only case I’ve heard of people running into issues is if you hit a kernel bug or depend on a really new feature that’s not present in the kernel yet.
Mixed workloads. Not everything inside MS is Windows, in fact, more than people probably realize is running on Linux. We have our own internal Linux distribution (Mariner) and own Kinvolk (which maintained CoreOS/Container Linux/Flatcar Linux).
That famous tech called cgroup was actually a Google contribution. But I agree that k8s is essentially Google's step to make themselves relavant in Cloud. They have missed the initial opportunity by promoting their PaaS AppEngine instead of something IaaS like ec2 in the beginning of the cloud competition, so Google just play the open-source game and keep releasing stuffs that can be used in all three clouds to lure people to use GCP. But then k8s is a very nice piece of tech that allows one to manage large clusters without vendor lock-in.
Cgroups, namespaces, apparmor/selinux, overlay filesystems, there is much more to containers than just cgroups.
The no vendor lock is looks great on paper but you are locked in day one. (Eg on aws you probably use IAM, LB, ASG for K8 Nodes - you can maybe move it to another cloud but the effort is going to be significant). Cloud agnosticism is a lie.
Effort will be significant for any global changes for non-trivial software. Significant effort is fine. Can you compare moving something from AWS to GCP for Kubernetes and for something like Lambda+Fargate?
I'd say not only is it a lie but is actively a not very good strategy to pursue right now, at least not all out as though you were pursuing some kind of multi-cloud end game.
At the K8s level, it seems like the introduction of the Gateway API is probably a good level of abstraction to work towards that will keep things about as flexible as possible without all of the insanity that comes with going beyond that to keep everything 100% vendor neutral.
It's very strange some of the phrasing of this post. I don't know AWS and Azure that well so I might be off here but every time I look over into those ecosystems it's hard to not come to the conclusion that GCP is literally several years ahead in terms of what they are offering.
Even just sticking strictly to the K8s space, there is nothing even close to matching GKE standalone, let alone GKE autopilot that I am aware of. In the world of serverless the gap looks to be even bigger.
I couldn't imagine a situation where I would even consider something else in a greenfield project / company honestly.
Do you know kubernetes or are you just jumping on the hate train?
Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
I know what it is, have worked with k8s, with borg and another half dozend similar systems.
Saying that you should use it because people know does not hold water. Also, anecdotally very few people understand k8s (setup and running it). Most people just know how to deploy apps on it (and even there most impressionable are young devs that haven't done things in a preK8s world)
I actually know k8 as well as another Borg-alike (Tupperware) that actually runs at Borg-like scale.
GP is absolutely right that it is insane overkill with no conceivable way to pay back the complexity investment in all but a few niche cases. Working at a FAANG isn’t the only niche where full-Kube makes sense, there are others, but it’s a niche thing, with the niche being roughly: I rack and run physical gear on a limited number of SKUs at “cattle not pets” scale.
If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust than any half-assed Borg-alike setup most people would come up with.
They’re happy to take your money if you yo dawg like imaginary computers so I put a docker in your Xen VM so you can Kubernetes while you pay me big bucks, but that’s because they’re not running charities.
I don't agree. The complexity isn't wildly different for complicated scenarios. Scenarios falling between the simplest and complicated absolutely benefit from running on a managed Kubernetes. Rolling your own Kubernetes is useful only at scales where you can't wrangle the complexity of self-service operations for developers, and even then it's still very questionable unless your headcount is astounding.
I would have agreed with you three years ago, but today there's a ton of surprisingly good off the shelf solutions, you can get very, very far in a brief amount of time with a single engineer. A ton of the functionality exposed by Kubernetes is absolutely necessary at any scale and very difficult to do yourself. "Don't let any process on this host talk to this port except these ones" is something that is so much easier in K8s than anything else I've ever encountered.
I feel that you’ve kind of talked past my point, though maybe I missed yours.
My point was that unless you’re running on metal, the cloud people built all this shit, and much, much better than k8s (or Tupperware for that matter).
They’ll stick you Docker container inside a VM happily because people pay them to do it, but it’s redundant.
Google didn’t OSS Tensorflow out of the goodness of their hearts, they did it to sell TPU hours on GCP.
k8s is likewise not charitable giving: it’s to sell managed Kubernetes on GCP, and subtle nudges to get people to make it their careers are uh, just business I guess.
Provisioning as code is in a sad state: Chef/Ansible/Salt/friends can’t deliver on idempotency, Nix is too complicated for most. So people use a container primitive (Docker) to work around that.
This has nothing, absolutely nothing, to do with service discover and failover and auto-grow and other Kube shit.
AWS does all that already. Better.
This is a borderline-cynical business strategy on the part of the cloud providers, and judging from the religious fervor around k8s (someone upthread has it in their username for Christ’s sake), it’s working.
This rings true every time I see someone talking about K8s. Companies just want something that is customizable to their workflow, they don't want a ready-made PaaS offering. Google's play was to provide an industry standard API interface that is available everywhere (which is essentially what Kubernetes is). Now that the control plane for compute is commoditized, GCP et al can sell complements - BigQuery, TensorFlow, GCS etc.
I think Anthos-like solutions are the future. Just buy the cheapest hardware from your local Datacentre or BareMetal providers like Hetzner and run GCP, AWS or some other flavor of software on top of it. This is also where Teleco Clouds are moving with OpenRAN and 5G, they are tired of being locked-in to full stack vendors like Nokia and Ericsson and want interchangeable solutions compatible through a common interface
Sure, there are better primitives within AWS for doing X, like running a containerized service, but k8s is more than that. Coupling to k8s is significantly more portable than a cloud provider's API. I can run the entire stack locally. I can run on GKE and EKS and my metal and your metal.
It's a powerful abstraction and it's very liberating.
I understand the cynicism, don't get me wrong, but that doesn't mean it's all bad. It definitely feels like a net positive to society even if at times it's unbearable.
Right, and GCP and Azure do it better on GCP and Azure. I've got no attachment to AWS, I just have to use it right now so it was the first one that came to mind.
Somewhere between "I'm small enough that all of the cloud providers are about the same" and "I'm big enough that I can leverage them against each other to get a better deal, but still small enough to not be on metal" there is maybe a sweet spot where I really do need to be mobile across providers in a push-button way. People in e.g. blockchain use k8s to good effect because they want to run a big distributed system with loose-ish central organization, and that's a real thing. It's also a niche.
I never said Kubernetes was useless full stop. I said that places where it's the right call are niche.
My current gig is humble, a usually-raman-profitable type garage band, we're like 5 engineers with Ethernet cables in our living rooms and a bigger AWS bill than feels fair, and if I proposed to any of the ex-FAANG type people involved that we should get serious about Kubernetes they'd laugh themselves pink in the face and resume typing. We can give our machines names and remember them all, there are a few dozen of them, but that's still "pets".
If we're fortunate and hard-working enough to need 1k machines, I'll spin up a ZK instance or something and write a little Nix/bash.
I won't be networking at KubeCon 2023 unless I lose a bet to one of my peers. I've got work to do.
Conversely I recently worked with a five-person company using Kubernetes to great effect with nearly a hundred customers. Being able to just commit a new .yaml to stand up or tear down resources is a real force-multiplier.
I do entirely agree with you, I think: there are too many companies using Kubernetes prematurely, but...
It's kind of turned out okay? Because of this, the ecosystem seems to be improving and it's easier than ever for small shops to get started. And selling software that targets Kubernetes is leagues better than targeting Ubuntu or Red Hat or specific kernels.
> I'll spin up a ZK instance
If I never have to run another ZooKeeper instance again I'll die happy.
I fully agree with you that the complexity isn’t all that different from the typical scenarios.
One of the scenarios I’ve seen Kubernetes replace is something like tying together Packer, Chef, Jenkins, Terraform, And Vault to get an EC2 instance propped up, but that’s just to deploy the code.
What was particularly annoying to me was that the developers were already using containers to build their code and another container for unit testing. So switching to Kubernetes would’ve been far easier. I had to go in and deconstruct the work they were doing with containers to then get their app deployed to an EC2 instance.
Although at this level you’re still not saving a ton by going the Kubernetes route. You get a ton of benefits with it, though. Even just tiny things like being able to get the config from a deployment using kubectl is so good. And then you can modify it, too! That’s been super helpful for troubleshooting.
I can confirm your viewpoint. I'm single engineer building self-managed k8s for my company. We're using OpenStack provider, but I wrote all the necessary scripts and it works just fine. We're starting to migrate to it soon, but all the preliminary tests were fine and I think it's completely manageable. I'm using simple kubeadm setup without any issues. I've read plenty of terrifying articles how k8s is hard but it's actually looks very easy so far.
I hope you take this as an admittedly weird compliment, but my lords and masters give me a few hours a week to code on my own shit and I'm trying to get a language model to talk like HN.
I'm going to try single-shot learning on one of the newer language models with "I can confirm your viewpoint. I'm single engineer building self-managed k8s for my company." because that remark, reasonable or not, so neatly encapsulates the current zeitgeist.
If you're trolling, respect. And if you're not, my 2c: flash an AMI and work on your product.
You seem to have missed the point of what makes Kubernetes a really nice system to work with.
It's not the machines. It's the comosible components, the service building blocks I can get with an API call.
* Want traffic into a cluster, ask for an Ingress.
* Want to auto-scale? Ask for a HorizontalPodAutoscaler.
* Want to manage CPU and memory rather than number of containers? VerticalPodAutoscaler.
* Want to point your monitoring system at a port? PodMonitor or ServiceMonitor
And of course, you can extend on the basic plugins. Want a bunch of databases? Maybe install the Vitess operator and simply make an API call to create an instance of VitessCluster. There are a dozen options, depending on what flavor of database you want.
Oh, and you don't need a sysadmin/SRE to do all this for you. You can be granted rights to a Namespace and compose everything you need in your namespace.
What commodity tool should I be replacing it with that gives me:
* super for heterogenous workloads (web servers, big batch loads, streaming workloads)
* autoscaling the workloads
* autoscaling the cluster
* auto-restarting workloads
* basically automatic networking
* the ability to have all that without the need to setup separate infrastructure, and to have the orchestrator move workloads around to optimise machine use
* the ability to hire people with experience in it already
* ability to run stateful workloads, and have disks managed and moved for you
* the ability to do all that by just providing a config
* the ability to do all that without having to let people have actual access to the machines/OS
* per container/workload topology constraints and requirements, that are network and data centre aware, without having to write any code
* have access to a wide range of tooling and docs
* be able to drop in workloads that automatically handle log collection, etc. without needing to configure each deployment.
* ability to swap out Network ingress/reverse proxy without having to redeploy or reconfigure your applications.
* ability to drop in any number of tools to do traffic shadowing, mirroring, splits, etc without reconfiguring or redeploying your applications
* ability to do all this without your devs needing to be networking experts.
* not locked to a specific cloud provider
I’m sure there’s tools out there that cover different subsets of those features, there’s probably tools that do all of them and more. K8s isn’t perfect, but it gives you an awful lot for some reasonable tradeoffs. Sure, if you’re deploying a single web-server/app, you shouldn’t bother with all this; and data centre level stuff probably has tools that make these features look like a cute picnic, but many devs I’ve worked with haven’t the capacity, ability or desire to figure that out.
> If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust
Sure, but you’ve still got to figure out how you want to setup, monitor, configure all of that, then stitch it together and maintain it, and also not get in the way of your main application development. To me, that’s what K8s gives me: I rely on the cloud provider to give me good machines, good networking, good storage primitives, and to heal/replace them when they fail, and K8s gives me a way to use all of that nicely, without needing to build and maintain it myself, and without my applications needing to know everything.
> How did people ever manage to do anything before k8s?
Nobody ever said it wasn’t possible. Nobody is saying “you can’t run things the way you want”, the argument is “you keep saying K8s is bad, here’s some reasons why it’s maybe not”.
> I'm going to make the claim that a load balancer + asg + basic networking setup is all you need in 95% of cases.
If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert.
> Learning how to package services and have them run anywhere? A lost art.
The issue isn’t my teams code-if we were only running our own stuff, and we only used a single language, packaging would likely be straightforward. But it’s the fact that we run several other applications, of varying degrees of “packaged” and the fact that I can barely get the Python devs to use a virtualenv properly that makes this statement a bit too unreasonable in my experience. Containers may be messy, but at least I can run 3 different spark pipelines and 4 Python applications without needing to dig into the specifics of “how this particular app packages it’s dependencies” because that’s always an awful experience.
> If you can make it support the autoscaling Spark jobs, the jupyter-hub env the analysts use, and all our GRPC API’s, and have everything mTLS’d together, and have all of DNS+cert + routing magic with similar setup and maintenance effort, I’ll convert.
Let's define what you're talking about here and I can show you the way.
What is the difficulty in autoscalling Spark jobs?
What jupyter env do they use?
You say grpc api. That's genericm what do those services really do? Sync request/response backed by a db? Async processing? What infra do they need to "work"
Where are you running your workloads? Cloud? Which one?
> What is the difficulty in autoscalling Spark jobs?
I mean, running spark is an awful experience at the best of times, but let’s just go from there.
Spark drivers pull messages off Kafka, and scale executors up or down dynamically based off how much load/messages they have coming through. This means you need stable host names, ideally without manual intervention. The drivers and the executors should also use workload-specific roles - we use IRSA for this, and it works quite nicely. Multiple spark clusters will need to run, be oblivious to each other, and shouldn’t “tread on each other” so the provisioning topology should avoid scheduling containers from the same job (or competing jobs) onto the same node. Similarly, a given cluster should ideally be within an AZ to minimise latency, doesn’t matter which one, but it shouldn’t be hardcoded, because the scheduler (I.e. K8s) should be able to choose based on available compute. Some of the jobs load models, so they need an EBS attached as scratch space.
> What jupyter env do they use?
We use jupyterhub wired up to our identity provider. So an analyst logs on, a pod is scheduled somewhere in the cluster with available compute, and their unique EBS volume is automounted. If they go to lunch, or idle, state is saved, and the pod is scaled down and the EBS volume is auto-unattached.
> That's genericm what do those services really do? Sync request/response backed by a db? Async processing? What infra do they need to "work"
The API stuff is by far the easiest. Request response, backed by DB’s, the odd analytics tool and monitoring tool as well. Servers themselves autoscale horizontally, some of them use other services hosted within the same cluster, all east-west traffic within the cluster is secured with mTLS via linkerd, between that and our automatic metric collection, we get automatic collection of metrics, latencies, etc. like what you get with the AWS dashboards, but more detail (memory usage for one), automatic log collection too.
> Where are you running your workloads?
AWS, but minimal contact - S3 is basically a commodity API nowadays, the only tight integration is IRSA, which I believe GCP has a very similar version of. So most of this should work in any of the other clouds, with minimal alteration.
I find it amazing how people take the absolute shittiest practice from the past and compare it to k8s and reach the conclusion k8s is better.
Have you ever done IaC where the whole copy of the infra came up on deployment and traffic was shifted between old and new? And developers could spin up their own stack with one Cloudformation or Terraform command? Have you used cloud primitives that solve most of the problems you are reinventing with k8s?
Yep, that's the typical straw-man from K8S zealots: compare static, even bare-metal setups with theK8S API "revolution", forgetting that the "clouds" (both public and private) already had APIs that covered most of it.
Sure, there are some nice additions on top of that like "operators" that take advantage of the main loop you have already perpetually running in a k8s cluster, but the leap is not that astronomical as some says.
If you think those APIs don't need tons of scripts to be useful to a company, then either you do everything through the AWS console, or you are an anti-k8s zealot yourself.
There are many established solutions on top of those APIs, that let you manage them with code, DSLs, graphical UIs etc. (i.e. Terraform, Harness, Pulumi, Spinnaker etc)
I have done everything from deploying to Tomcat servers to modern k8s. I really don't like Cloudformation and ECS, arcane and poorly documented syntax. I like much more having a Helm chart for my apps. Have you ever tried to set up private DNS in ECS for services to talk? That doesn't look simple (or well documented) at all to me.
By the way, Terraform and k8s are not exclusive, I use them together.
I'm right now migrating from "anything before k8s" to k8s.
I can tell you how we managed to do anything.
We had few servers. They run docker-composes scattered all over people's home directories.
We had one nginx entrypoint. It's forbidden to restart it at work hours because it won't restart and there will be hours outage. You expected to restart it at weekend and then spend hours trying to start it.
Some docker images are built locally.
Backups are basically not worked before I came to this company. It was the first thing I fixed. Now we can restore our system if server dies. It'll take a week or two.
Kubernetes is going to be sane exit.
Yes, I know that it's possible to build sane environment. Problem is people don't bother to do that. They build it in the worse possible way. And with kubernetes, worst possible way is much better because kubernetes provides plenty of quality moving blocks which you don't need to reinvent. You don't need to reinvent nginx ingress with all the necessary scripts to regenerate its config, because you've got ingress-nginx. You don't need to invent certbot integration because you've got cert-manager (which is my personal favourite piece of k8s). You don't need to invent your docker-composes storage conventions, user permissions (777 everywhere, right), because k8s got you covered. All you need is to learn its conventions and follow them.
> Do you know kubernetes or are you just jumping on the hate train?
It seems rather that you don't know what kubernetes is. His comment is justified. You can put your pitchfork back.
> Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
It seems that you don't understand ehat he's trying to say.
>It seems rather that you don't know what kubernetes is.
The company, for which I helped migrating to k8s, pays me 6 figures to keep maintaining it, thinks otherwise (also look at my username ;) ). All of our developers love it because it replaced some weird hacked together deployment stuff.
Which part do you think he is right or wrong?
It just seems that in every thread people moan about k8s. But it seems that it's usually people who are overwhelmed by it, which is a legitimate reason not to use it. But just saying that it's generally bad is what starting to annoy me a bit.
That was pretty much my experience. From a distance, it looks over-engineered. Once I started using it in my personal projects and at work in production, it makes a lot of sense and it’s far easier than people make it out to be.
I’m currently helping my employer switch everything to kubernetes and we’re replacing a lot of really, truly over-engineered trash.
I will say I do hate yaml but there aren’t a ton of choices, tbh.
Hi - since you're on credentials, I hold CKA, CKAD, and CKS.
I have spent my entire Kubernetes career unfucking the people three degrees removed from you who hear advocacy like this and say wow, yeah, the people with gripes about Kubernetes are just moaning and overwhelmed. Meanwhile, a team at an unnamed government agency just allowed Accenture to con them into turning multiplayer Excel, a process which used to take two humans about a week, into an $18 million annual AWS system funded by the U.S. taxpayer. Because of choices-du-jour, including Spark and Kubernetes.
It just seems that in every thread people moan about people moaning about Kubernetes and then project their interpretation of what that moaning is after starting from the presupposition that it's moaning. If you'd instead listen to the other side instead of the basically-crypto-advocacy-train-choo-choo of "if you have gripes, you just don't get it," you'd understand the cons a little better. Such as my entire decade of professional Kubernetes consulting being every single computing project, including in the Fortune 500, the Department of Defense, and at second-tier VC startups in the Bay Area, being a complete operational disaster that's bleeding capital like it's free.
Capital is no longer free. Read the news.
I now wield my certifications to advise against Kubernetes in any capacity because it is a strategic risk unless resourced exactly right and funded exactly right (which is impossible to justify on infra budgets), simply because of the experience I have watching people like you come in, explain to leadership that it's fine, and then ride that salary until it doesn't suit you anymore and leave a system that takes 5-6 FTEs to comfortably administer on whoever gets to pick up the ball. For every comment you see on Hacker News where someone like you is saying "yeah, Kubernetes runs great for us," they're (a) leaving out the operational toil, particularly in the break-fix department where the entire Unix paradigm changes and (b) not realizing that there are five other teams like them that are completely underwater while trying to put out their overengineered tire fire.
And remember, nearly every project using Kubernetes is trying to do multiplayer Excel. That's it. And the more we invest in trying to make the world's most overcomplex computing system do that use case even harder, the less we think about "isn't there a simpler way to do all of this from first principles?"
I am currently overengineering my previous work to run it inside kubernetes. It isn't my decision, i just asked for a VM and the access to ansible, but the chiefs wanted to use kubernetes for everything.
I don't think this is that bad tbh. I runs my unit tests on container construction and while it was tedious to develop, mostly because i cannot run docker on my work PC, i understand that it makes it easier to inventory and check the advancement. The management will always love it, so i think kubernetes is here to stay. Not because it solve a lot of engineering issues (not for me anyway, i'd rather run a cron in my VM than declare a kube cronJob), but because it solve a lot of other fondamental issues for the management (inventory, visibility on existing/running projects. And everybody use the same system without using the _same_ system.)
I don't know about your team, but in our team I try to move as much as possible to kubernetes so I have a common way of deployment across all members. Of course if you are just thrown into the cold water without help from the Kubernetes guys, it will suck.
Also I noticed that once developers "get it", they actually like it a lot.
Our k8s based deployments go so much smoother than our VM based deployments (Ansible). We have centralized logging set up for k8s, but not VMs. We have post-update health checks for k8s pods, but not systemd services. Et cetera.
I understand that. But i already have the code to set up logging and health checks (i basically took the code i did at my previous work, it was open source so no issue here).
Like i said, it is fine, i'm not mad about it, it took some time to understand but i get it now and while still less efficient, i'm way better at it, and when i finish this project the next one will be easier. Also, it is important that the whole team use the same product: it was a good reason to loose one man-month to get me up to date with the rest of the team and facilitate my next "app".
I've always said that it reinvented the mainframe by turning it inside-out. I did some work on an IBM 360/370 running VM/CMS and I can see a lot of parallels in how both systems operate. Most of the architecture is an interface that is implemented by some pluggable piece of software and there's also all of the resource scheduling and management which is a first-class component. This makes the system very flexible and reliable.
And I think k8s is here to stay. If what you have is a bunch of containers you can easily push that into k8s. It's not any more complex than any other solution you would use to deploy them.
Let me tell the truth that I think. I'm k8s newbie and I'm delighted by it and terrified by it at the same time.
I really like k8s architecture. It's just miracle architecture that I'd never come with myself. It's very extendable. I'm very inspired by it and I hope to apply it to my own projects some time.
But UI is terrible.
scopeSelector:
matchExpressions:
- scopeName: PriorityClass
operator: In
values:
- middle
Like really? I thought parsing problem was solved like 50 years ago. Why not `scopeSelector: "priorityClass in [middle]"`. It seems that we forced to write YAML AST instead of proper programming language. It really seems like some intermediate form that's supposed to be output from a proper frontend language but we've stuck with it.
Also I'm not sure that I like all this modular nature of k8s. Sometimes it makes sense, for example for CSI. But why should I choose between CNI implementations? I just want best one and that's about it. I want it to be from Google. I don't need to choose between kube-apiserver implementations or kube-scheduler implementations, thanks god.
Also the whole kubeadm story seems half-baked. It works, but far from ideal. I guess it'll get better with time.
But those are small complains. Like I think that it has a good potential to improve. But there's nothing to replace it. Even if someone will build some lambda swarm which is better than k8s, nobody's going to migrate to it, because k8s is offered by so many cloud providers which made investments and are not going to move to other solutions in the foreseeable time.
> It really seems like some intermediate form that's supposed to be output from a proper frontend language
Honestly, it is. You're supposed to write your own operator that would in turn generate all this boilerplate properly — you can look at a tech blog of any large-ish corp that tried to use k8s, and they all say "yeah, we tried to write those manifests manually, it was horrible, we now have a home-grown piece of software that generates it" (I mean, don't you like it that your deployment manifest must have metadata.labels, spec.selector.matchLables, and spec.template.metadata.labels all have exactly the same contents or else strange and wonderful things happen?)
Then again, some people make their operators' input language way too flexible so in the end you again have a YAML-serialized AST of an accidentally Turing-complete language... but it's the common ailment that plagues us all: given a language, any language, instead of using it we start building an interpreter for some other ad-hoc language on top of it, and then suffer from the consequences.
As for standard k8s' operators... they're okay, but the moment you need to do anything slightly different from the intended way, you're in a see of hacks and workarounds. For example, you want your pods spread perfectly even between a set of nodes? Haha, why'd you want that, the k8s cluster is supposed to be RAID-like but okay, here's the beta-feature called topologySpreadConstraints. Watch out though: it disregards any inter-pod affinity constraints, and you really want to use it with a nodeSelector/nodeAffinity.
What k8s offers is a platform, on which people can build their own solutions. It makes solving a lot of very hard problems a lot easier and portable. Before k8s, tools like these were out of reach for small/mid companies with only a handful of dev-teams, but enough for organisational scaling to become a problem. K8s allows self-servicing while still enforcing standardisation.
And I'll give you another use-case I'd like to see you replicate with any other existing platform: migrating an entire on-prem application stack of 200+ services to the cloud. It took us 3 weeks total to do for a client of mine, where the hardest part was the migrating state with minimal downtime, and a lot of work went into the IAC part for the GKE setup.
Also, institutions like banks have now jumped on the k8s (well, mostly openshift, but same thing) train, I'm pretty sure it's here to stay.
Not the biggest k8s fan out there but I think the verdict is that K8S will stay for a looong time, especially in the enterprise that it's just starting adoption now.
Yeah I’m not sure if it’s the Reddit mentality at play here or actual outage but it just always seems like these types of comments are reacting to a headline, it is unfortunate.
Here is the fundamental problem. You cannot explain (a lot of people worked with containers and kind of get what they are but nit really) what a container is and point to the article. I've read the article and it confidently talks about docker containers. I can tell you for a fact that just because you're using a docker file and build a image does not mean there is anything special about the container that docker spins up. In fact there is no such thing as a container. We made it up. It's just a bunch of Linux kerned features that are used together creatively.
A container is not a replacement for properly securing/isolating your app. Any flaw in any of the mechanism the kernel uses to provide the "containerization" features can end up with all the containers on the node compromized. Also, unless you're really disciplined about how you're building your images you probably already have a bunch of zero days lurking in your deployments.
It's all fun and games until we actually need accountability and the buck has to stop with someone.
"Containers" are just a pattern, the real technology behind it is kernel namespaces (plus ordinary process boundaries). (For that matter container security is still half-baked in many ways. The kernel namespacing has been added after-the-fact and it can't really be trusted to make an effective "sandbox" on its own.)
Everyone keep saying that. But containers are battle-tested for years at this point. I think that all low-hanging vulnerabilities have been fixed. Like you can expect that your container IS secure if your system is up to date. Every container escape is kernel bug which is fixed ASAP. And 0-days can be applied to everything, virtualization will not help you with that, there're 0-days in virtualization layers. Yes, surface of attack is significantly larger with containers and chance of 0-day is larger, but for most companies which don't really run code from attackers in their containers, this danger is over-stated. Container security is good enough for most workloads.
Well, not so fast. Ubuntu apport (its coredump collector and crash reporter) had a series of bugs over multiple years which could be used to escape from containers. This is all because coredump filter is still a global thing and not a per namespace one.
I agree with you it's actually more secure than people thought it is. A rule of thumb: just treat it as same as "running multiple applications with different user". Yeah, not a real sandbox. But nowadays also not many low hanging fruits, either.
Yes, and that "everyone" include kernel developers. That's my point. Now, does this mean containers are totally useless, of course not. Especially if you can prevent running attacker code via other means.
I’m not too proficient with containers so maybe I am ignorant, but I view containers more like a package manager.
A proper golden image pipeline can work well, and you can rightsize your workloads easily.
All of the additional features such as ingress controllers and other complexities make debugging a bit obtuse, especially if you are not extremely proficient.
I could be an old curmudgeon but simplicity is my priority.
I think its vastly more useful to think of a container as an RPM with layered tarballs, a chroot/jail, and some cgroup isolation.
Thinking of them as a virtual machine is the wrong metaphor.
Kubernetes replaces the thing we were doing in 2003 blasting around tarballs of software with scp and some scripts to start and stop things, and is the end result of evolving from that point to a much more dynamic one and with cgroups providing the ability to have software teams not quite clobber each other's workloads while running on the same hardware.
The problems it solves overlaps with many of the problems that VMs were addressing, but it is like looking at two different species that look quite similar but are in completely different animal families and are just similar because of the similar environment they evolved in.
How ingress controller makes debugging obtuse? You're using nginx in front of your services anyway, right? Ingress controller is just nginx with some tool autogenerating its config from kubernetes objects. You can view nginx logs easily.
> I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered
Have no fear, these k8s specialists are going to be extremely in demand for getting rid of legacy k8s hell. (Not even joking, I'd pay good money for someone to get rid of the k8s cluster previous consultants have cobbled together.)
I know most configurations don't use it, but k8s supports different runtimes and today you can use Kata or firecracker to run each pod in it's own "microvm."
You then get the security benefits of a "real" VM while still distributing your software as an OCI (Docker) container, and can operate it like any other k8s cluster.
Agree. As someone that has had to migrate systems from K8s to Elastic Beanstalks or equivalent, there are definitely situations where K8s is overkill and simpler systems can work better.
A lot of comments here about the complexity. But a takeaway from the article is that driving a car is different from repairing it.
We are a startup, had zero knowledge of k8s before. Without GKE, we would have lost days and days of work to imitate parts of k8s on VMs.
Scaling up and down, everything as a container (even cron jobs. Otherwise we had to keep a vm running and use some scheduler).
GKE has been a massive advantage for us.
The complexity bits may be true for someone managing a k8s cluster. Even with just 3-4 services, managing everything as cattle has been amazing.
To be perfectly fair with you, GKE is another level from every other k8s installation target.
EKS and AKS are not nearly as well polished and put together, and god help you if you decide to run on-prem.
The problem is of course that you’re quite right that it’s like driving a car, but ultimately unless you have a good support contract with your car manufacturer for a specific use of the car, then you have to take it to another mechanic and they’re going to put you over a barrel.
Hosted k8s is a blessing for development teams. A great benefit is that integrating k8s into tools such as Github or Gitlab pipelines makes it a unified and easy deployment cycle. No more issues regarding different release procedures while someone is out of office.
I couldn’t agree more. This and the cloud foundation toolkit for terraform is why we chose GCP, and it got us up and running with an impressive infrastructure in a few days.
Is your goal really to be up and running in a few days? My goal is to run for years without significant ops effort. Maintainence cost always, always dwarves development cost.
> On the darker side, inefficient scaling (due to uneven daily/yearly traffic distribution), overly complicated deployments (delivering code to many boxes quickly is hard), and fragile service discovery (have you tried running consul or zookeeper at scale?) would lead to higher operational costs.
Obviously, everyone's use-case is different but none of this has been my experience with VMs:
- Both AWS and GCP allow you to define custom scaling policies for an instance group.
- Deploying new code just means updating a template with your new image and pushing that out. Both GCP and AWS would then cycle the existing machines automatically to use this new image.
- At least in our case, there is no need for Service Discovery; you just need to know where the Load Balancer is.
We have moved some things to Kubernetes and now have to deal with inexplicable occasional failures. That and an everyday massive YAML headache.
narrator: it is not. you're not trying very hard if you haven't hit a case where k8s will lead to you banging your heads on any available walls trying to understand a behavior in an edge case (here is one that made walls hate me: https://github.com/kubernetes/kubernetes/issues/96858 and I know of a half dozen developers that learned about this the hard way).
Looks to me like an example of aws not having good defaults put of the box and not an k8s specific issue. Having to specific a `sleep 30` shutdown hook on pods is quite annoying though and some built-in should be added instead.
You're missing the point. There is no cloud agnosticism and you need to understand the underlying cloud primitives although allegedly k8s isolates you from this.
I'm not missing the point. The OP was specifically talking about a k8s bug that wasn't a k8s bug at all. You don't blame Linux for a poorly written application. If the cloud providers choose to add a bunch of uncommon options that has no counterpart in k8s that's their choice. but it doesn't mean k8s is a magic bullet that makes it so you don't have to do the bare minimum to understand the cloud providers.
How do lightweight MicroVMs which have the speed of containers but the isolation of a real vm, as implemented in FIrecracker etc, change the landscape? Or do they? I imagine launching a k8s pod or container is still faster and offers much more management and the security benefits of a real vm arent really applicable or desirable?
the revolution will start across this dimension at some point in time. We're looking at unikernels (https://en.wikipedia.org/wiki/Unikernel) and boot times of nanoseconds for these super specialized VMs. The tech and the ideas around it are not quite there but this is an exciting space to watch in the next 5-10 years.
That is so close to a simple process in a multitasking OS ... I get it, you still need dependency isolation, you're just reducing the dependencies to the minimum number as possible, but sometimes I wonder if something (besides cgroups) was missed in OS design.
Launching a container is implementation detail for Kubernetes. I could totally see Kubernetes which runs some pods in containers and some pods in VMs with tiny kernel. May be it even exists, I'm too lazy to google it.
Microvms are just the workloads, while Kubernetes is a platform, mostly agnostic of that workload. Yes that's currently mostly containers, but you can already schedule VMs on k8s with KubeVirt.
Most of the comments here are missing the point of the article.
Yes, Docker containers != VMs. That's the point: An old-school VM had all the components needed to run one instance of a service; this might include the core app itself, along with "support processes", like a reverse proxy for TLS termination, and maybe some other services. By design, you cannot run all of those in a single docker container.
The point of the article is to say that Kubernetes pods ~= VMs: A pod has all the components required for a single instance of a service, each one in its own docker container, but with increased access to other containers in the pod (just as processes have increased access to other processes in the same VM). So if you're coming from a VM deployment world, when you see "pod" think "vm", and when you see "sidecar" think "support process".
Note that I said "Docker container". And that's an oversimplification, but from the article:
> Docker containers have been deliberately designed to contain just one application. One container - one Nginx; one container - one Python web server; one container - one daemon. The lifecycle of a container would be bound to the lifecycle of that application. And running an init process like systemd as a top-level entrypoint was specifically discouraged.
My guess is that the goal of this is to leave "lifecycle" issues at the container boundary. If you put nginx and your golang webapp in the same container, and one of them crashes, Docker / kubernetes you have to have logic inside the container to detect that and restart the correct one. If the goal is to have Docker / Kubernetes is to take care of all that for you, making it one thing per container is the best option.
> If you put nginx and your golang webapp in the same container, and one of them crashes, Docker / kubernetes you have to have logic inside the container to detect that and restart the correct one.
If the proxy only serves the app you can just shrug and restart the container. If the proxy and the web app are both stateless it shouldn’t matter.
What you don’t want to do is mix stateless processes with statefull services, or have intertwined dependencies.
I'd rather use infrastructure to package orthogonal concerns than to figure out how to shoehorn all my concerns into each of many container builds. Keep the containers simpler, no?
This also makes containers more dev friendly, dev accessible.
It’s astounding how long companies can ignore this about their people. Acknowledging that kind of power in an employee probably messes up the power dynamic in uncomfortable ways.
I recently learned that one of the heads of SpaceX is heading his own company now. It’ll be interesting to see if his bus number was high enough that the wheels stay on, if the symptoms will be obvious if it wasn’t, or if they will be explained away by other phenomenon. I have had enough successful bosses that credited the wrong actions that it stands to reason that most failures are also blamed on the wrong chain of events. I know my own narratives in 5 Why’s situations are often in stark contrast to other theories.
Define 'fast' and 'these days'. Lots of companies are affected by bus problems, but I'm not so sure they're successful at dealing with it. I'm also not sure this is a new phenomenon, and it's certainly not universal. If anything I can think of a couple of projects that suffer from too little turnover, and the resultant echo chamber.
As for 'dealing', look at all the status Google services have lost in the last ten years due to systemic problems with valuing novelty over stability. There are perverse incentives not to maintain things you have delivered, even if you are still at the company.
That doesn't make sense? K8s runs Docker images, and a Docker image explicitly is not a VM but, in fact, a lightweight replacement for a full-blown VM. Of course, running Docker on Windows or Mac OS requires a Linux VM, though.
Kubernetes reinvented virtual machines? That’s a bold statement. I’m still waiting for them to catch up to where OS/360 was 45 years ago.
People forget that mainframe did it first, mainframe still does it best, the only thing that’s changed is that now you can do it on consumer grade electronics for less then 5 million dollars.
The containerization features used by projects like Docker and Kubernetes are provided by the Linux kernel. Containers are not VMs, they're just regular processes.
They are regular linux processes, you can find them in /proc or display them with ps... Croups, namespaces etc are Linux features that can be used with any process.
But i guess i see now what you are saying. That it's a "meta" term for that idea of abstraction.
K8s can actually run VMs as well (kube-virt) so you can theoretically align on a common platform for both VMs and Containers. In practice most people just use their cloud provider's implementation of K8s. It's complex but if I had to run on bare-metal I'd probably look in that direction vs playing for an equally complex proprietary solution.
Does anyone know of a good, sober treatise on the suitability of microkernels for multitenancy and/or split priority systems?
Hypervisors just seem like particularly impoverished microkernels to me. It reminds me of social programs working hard to avoid the term Socialism because of emotional baggage. Oh this is definitely not a microkernel? Is it a full privilege program that handles a small number of system-wide concerns and delegates the maximum number of other concerns to processes running at a lower privilege level? Yes? Then can we call it a… no? Okay buddy.
> At the last HotOS, Mendel Rosenblum gave an ‘outrageous’ opinion that the academic obsession with microkernels during the past two decades produced many publications but little impact. He argued that virtual machine monitors (VMMs) had had considerably more practical uptake, despite—or perhaps due to—being principally developed by industry.
> In this paper, we investigate this claim in light of our experiences in developing the Xen [1] virtual machine monitor. We argue that modern VMMs present a practical platform which allows the development and deployment of innovative systems research: in essence, VMMs are microkernels done right.
(Edited to add relevant paragraphs from the abstract.)
Most people use kernel to mean "operating system kernel" only. Your definition is a superset of that, so it fits the general definition but then expands it in a way that doesn't have broad consensus. That's why people disagree with you.
That's the tragedy of software development. People like complex things because it shows how smart they are and how hard the problem they are solving really is.
Good for some use cases, but still plenty of servers that are hostile to K8s because ie you have to login and enter a license key, or something like that.
I don't need to rewrite my apps in order to run them on a different architecture. Recompiling is enough for compiled languages, and for interpreted languages you just need to add the interpreter appropriate to the architecture.
I suppose you missed that it was a bit tongue in cheek but if you want to take it seriously, I think it still stands. You can just compile to JVM bytecode instead of the assembly of your vm of choice. It's indeed different in practice but I don't think it's all that different in principle.
Really though, I was just pointing out the tautology of your previous comment. It boils down to "it doesn't run anything that doesn't run on it."
> I suppose you missed that it was a bit tongue in cheek
Indeed I missed that! :)
> You can just compile to JVM bytecode
Perhaps, but it's a hack with no clear advantage, whereas with a container I'm just using kernel features for better isolation and resource management. The application was compiled the way it was meant to be compiled when written.
> the tautology of your previous comment. It boils down to "it doesn't run anything that doesn't run on it."
Perhaps if you take my comment in isolation, but when you put it in context it doesn't.
The thread went:
- Kubernetes reinvented VMs in a good sense (OP)
- I thought we had this kind of covered 20 years ago with J2EE (a rebuttal of OP)
- Doesn't really handle multiple process isolation and IPC as well. (your own comment rebutting the rebuttal :P )
- It also doesn't really handle anything that doesn't run on the JVM. (my comment, adding that it's even farther from the J2EE)
Hopefully this shows how my comment, when taken in context, is not a simple tautology because it adds information.
