I run stuff on hosted k8s from DO, Google and Vultr. I can absolutely reuse my knowledge, and deployments are almost identical (minus smaller differences like storage csi driver, etc)
I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP). I can assure you they are radically different; autoscaling works differently, the load balancers work differently, the networking infrastructure is completely different, the failure modes and limits behaviors are different, the instances perform differently, observability is different, and they all suck in unique and different ways that we discover on a daily basis. Shit even AWS can't keep their regions consistent; each region has different products and features and they fail in different ways.
If you are the one maintaining it it's a full time job handling all these edge cases, it's completely miserable and I wouldn't recommend it to anyone.
> I work at a place running a million containers deployed in all 3 (Azure, AWS, GCP)
Are you using AKS, EKS, GKE on those providers, or deploying your own k8s on top of the compute those providers offer? It sounds to me like the former.
I’ve done smaller deployments on GKE and another on EKS, and I can tell you, they are different enough. It’s when you start having to autoscale, optimize resources by instance types, and manage network ingress that these quirks start really come out. The essential ideas are invariant across cloud providers though.
I should’ve been clearer about what I was getting at. I agree AKS, EKS, GKE etc (cloud gnostic k8s) are different enough to cause a growth of complexity when managing a mixed environment of them.
The post I was replying to seemed to be saying (by analogy) “Linux is hard to manage because I run into all sorts of trouble trying to support a mixed environment of SuSE, Ubuntu and RHEL, therefore Linux is just too complicated”.
The essential ideas that Kubernetes exposes concretely are invariant across cloud providers. There absolutely are nuances and quirks that are different for each cloud provider, and unique for the workload you have. However, those same ideas also act as a kind of mental framework in which these quirks can be understood from. It isn’t as if those quirks are randomly there, unconnected to anything, and therefore not part of a coherent design.
For example, the consistent use of labels as a way to identifying groups of resources that need to coordinate with each other is very useful for any distributed system. I find myself looking for them in say, CI/CD systems (in the form of agent tags), or at the application level in say, matching players to game servers.
Some part of it creates cloud specific resources, and you might also for good reasons have cloud manged database or data storage that your k8s services use. However, "k8s on a specific cloud", is mostly the same, except for the outer edges.
I enjoy working with Kubernetes, but forcing a complex domain into something legible is a recipe for catastrophe. There are quirks, across cloud providers, and this is just another day in Ops, with or without Kubernetes. (See: https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call... )
No you don't have to. You can deploy your own cluster instead of using the managed option if you want to. A good SRE can deploy and manage EKS. A great SRE can deploy and manage a cluster to any Cloud without ever touching the dashboards.
