Managed solutions make K8s easy to use, while we can still benefit from being able to run out workloads left and right on any cloud vendor. In one word: portability. Which in the day and age of cloud vendor lock-in it is to be protected at all cost.
I know that some organizations allow to allocate time to explore new technologies and learn new practices. If your organization has no policy around this, it is worth try asking. Ultimately it will benefit the organization and the business as a whole, as they will be able to build a solid foundation to more rapidly transition and execute on their digital products. Kubernetes is good for business.
All this knowledge is still relevant. Kubernetes is an extra layer on top of all the classical technologies out there. It is not a replacement - besides some fancy experiments, it doesn't replace GNU/Linux.
If you're going to run a Kubernetes cluster, you still have to know all the "classical" networking, GNU/Linux system administration and architecture, etc. Having a fancy task scheduler for containers, and a tool that sets up clever network bridges (or whatever your preferred CNI flavor does), and a bunch of other nice extras does not remove the requirement to know what's underneath.
And in any non-standard situation, there are chances that you'll have to build things from scratch. For example, I do want to run a K8s cluster over a cjdns overlay network (which doesn't have IPv4, so Flannel or Weave won't work). Haven't figured this out yet.
And if you're not setting up and managing the infrastructure - it's about the same as it had always been. Just a different user interface to manage your deployments/storage/networking.
> Managed solutions make K8s easy to use
I would disagree. "Using" is the same (that's the whole point of K8s). "Install and maintain" is easier, but only in a sense that you don't do this. ;)
> If you're going to run a Kubernetes cluster, you still have to know all the "classical" networking, GNU/Linux system administration and architecture, etc. Having a fancy task scheduler for containers, and a tool that sets up clever network bridges (or whatever your preferred CNI flavor does), and a bunch of other nice extras does not remove the requirement to know what's underneath.
This so much. I went from one year as a junior HPC cluster admin (glorified title, I was a software builder and then focused on container usage) to a 6 month internship where I was focused on being part of an OpenShift team. I’m fairly good am maneuvering my way around a system and getting things working, but being thrown head first into that I realized how little I actually understood about systems, particularly networking related. I didn’t have a lot of time, and I learned a lot about OpenShift and K8s in general, but I felt more like an advanced user who could explain things the others trying to learn their way around and build small tools rather than an admin of the platform. Maybe I’m selling myself short and experiencing imposter syndrome mixed with being dumped into huge, pre-existing, and foreign infrastructure, but it was an eye opening experience.
Since that’s ended I’m at a new gig as a “standard” sysadmin. I’m using this to skill myself up and take the time to really understand as much of the layers and how they work together as I can, both on and off hours.
I’d love to get back into the K8s area, it’s such a fascinating workflow and paradigm, but I have some personal progress to make first.
However I do agree on vendor lock-in. At least to a degree. KVM and Xen didn't really prevent this for cloud environments.
I do think the main takeaway from a lot of this is the way software is designed. It is becoming more self-contained. Docker in many respects is like a static binary. FatJARs are a similar approach. Also Go in general seems to go that path.
What Kubernetes really does is providing an agreed upon standard for infrastructure, similar to what Docker gave for software packages.
They enabled concepts like Unikernels to at least become interesting, because they smoothed the way for the thinking of software that should be self-contained.
I think the future really is one where Kubernetes and Docker are just annoying overhead, where we find it odd how something "emulates" them, just how terminal emulators emulate... well, terminals.
We are in a feedback-loop where we put a more and more tight corset on the software we develop. First there were compute clouds, where developers learned it's bad to keep state around, then there was Docker, then Kubernetes where certain best practices, that have been best practices for a long time "forced" them to be followed more and more, especially because whoever provides your infrastructure and the developer are able to agree on the interface.
Docker and Kubernetes are standards due to their dominance, similar to Internet Explorer back in the days and Chrome today. As of now there is only minimal written specifications. Most of it is standardized by the implementation. Hopefully this will change some day to stabilize the interface and give opportunities for competing implementations, so more innovation can happen outside the boundaries of these projects, allowing for competition.
Maybe this has a positive influence on complexity as well.
K8S doesn't eliminate the workflow for "deployments, upgrades, monitoring, etc.." it just black boxes them. It also assumes out of the gate that everything needs to be able to do HA, scale for 1,000,000 instances/s etc...
Over and over and over people show examples (I'm guilty too) of running internet scale applications on a single load balanced system with no containers, orchestration or anything.
So please stop preaching this as something for general computing applications - it's killing me cause I've got people above me, up my ass about why I haven't moved everything to Kubernetes yet.
Kubernetes does not black box anything. At most it abstracts the computer cluster comprised of heterogeneous COTS computers, as well as the heterogeneous networks they communicate over and the OS they run on.
I'm starting to believe that the bulk of the criticism directed at Kubernetes is made up by arrogant developers who look at a sysadmin job, fail to undertand or value it, and proceed to try to pin the blame on a tool just because their ubriss doesn't allow them to acknowledge they are not competent in a different domain. After all, if they are unable to get containerized applications to deploy, configure, and run on a cluster of COTS hardware communicating over a software-defined network abstracting both intra and internet then of course the tool is the problem.
Why not? The Kubernetes/serverless/DevOps people have a compelling argument--organizations can move faster when dev teams don't have to coordinate with an ops/sysadmin function to get anything done. If the ops/sysadmin/whatever team instead manages a Kubernetes cluster and devs can simply be self-service users of that cluster, then they can move faster. That's the sales pitch, and it seems reasonable (and I've seen it work in practice when our team transitioned from a traditional sysadmin/ops workflow to Fargate/DevOps). If you want to persuade me otherwise, tell me about the advantages of having an ops team assemble and gatekeep a bespoke platform and why those advantages are better than the k8s/serverless/DevOps position.
IMO, some of these practices come from businesses with huge rivers of money who can hire and retain world class talent. I’d like to see some case studies of how it works when your tiny DevOps team is spending 80% of their time managing a huge portfolio of small apps. How then do you deliver “new, shiny” business value and keep devs and business stakeholders engaged and onboard?
Come kubernetes, and we have a way to blackbox developer excesses, push 12 factor onto it, and generally out of over 60 present apps, we have reduced our workload to really caring about maybe 5 classes of them, as they are commonalized enough that we can forget them most of the time.
At different job, we're pushing heavily towards standarized applications, to the point of Ops writing frameworks for the devs to use - thanks to k8s we get to easily leverage that, compared to spending lots and lots of time on individual cases.
> your tiny DevOps team is spending 80% of their time managing a huge portfolio of small apps
In a DevOps world (the theory goes), the DevOps team supports the core infrastructure (k8s, in this case) while the dev teams own the CI pipelines, deployment, monitoring, etc. The dev teams operate their own applications (hence DevOps), the "DevOps team" just provides a platform that facilitates this model--basically tech like k8s, serverless, docker, etc free dev teams from needing to manage VMs (bin packing applications into VM images, configuring SSH, process management, centralized logging, monitoring, etc) and having the sysadmin skillset required to do so well [^1]. You can disagree with the theory if you like, but your comment didn't seem to be addressing the theory (sincere apologies and please correct me if I misunderstood your argument).
[^1] Someone will inevitably try to make the argument that appdevs should have to learn to "do it right" and learn the sysadmin skillset, but such sysadmin/appdev employees are rare/expensive and it's cheaper to have a few of them who can build out kubernetes solutions that the rest of the non-sysadmin appdevs can use much more readily.
k8s makes no assumptions about your workloads, it just gives you tools. And it's super useful even if you don't need to do HA or scale to a million instances.
Most production apps still need to manage deployments and rollbacks, configuration, security credentials, and a whole bunch of non-scale related things. And k8s makes a lot of that significantly more manageable.
Of course, this is overkill for a single application, but as you start adding more applications that need to be managed, the benefits really start to add up.
If you give something like GKE a chance, you might be pleasantly surprised. :-)
What I've seen, anecdotally, is that many ops-background people don't "get" why kubernetes is such a big deal. They assert rightfully that they can already do everything, they already know how to do everything, and they can do it without the overhead (both cognitively and in terms of resource utilization) of k8s.
But, if you are writing and deploying code - especially if you're not in a terribly agile organization - k8s eases so many real pain points that "old" models have which ops teams may be only vaguely aware of. If you need a certain dependency, if you need to deploy any new software, an entire new language or approach, if you need a new service, you now have the ability to directly do it immediately.
I can't tell you what a big deal it is going to be for a developer at a random bigco to be able to run their code without waiting for ops to craft a VM with all the right bits for them.
k8s solves real problems. If you have a monolith and need to solve how to scale it, that's not where k8s shines. But with lots of small workloads, or dynamic workloads, or existing dev vs ops organizational hurdles, it can really be a game changer.
Quite the opposite. k8s isn't easy to set up, run or maintain. A large company running clusters with millions of nodes is probably more capable of letting it appear smooth than some small hoster with only a few servers.
No, it isn't. It's extra complexity most don't need.
Although I would argue that you need to know what trade offs you are making if you have the right use-case (multiple containers you need to orchestrate, preferably across multiple machines) and you are not using it or a similar tool. There are lots of best-practices and features you get out of the box, that you would have to implement yourself.
* Deployments and updates (rolling if you so wish)
* Secret management
* Configuration management
* Health Checks
* Load balancing
* Resource limits
And so on(not even going into stateful here), but you get the picture. Whatever you don't get out of the box, you can easily add. Want Prometheus? That's an easy helm install away.
Almost every system starts out by being 'simple'. The question is, it going to _stay_ simple? If so, sure, you can docker run your container and forget about it.
I've been holding off setting up a system like Spinnaker because I'd read it was coming (in the form of custom deployment strategies), but can't find anything current on the subject.
At an application level then the strategy consists of having two applications deployed and update the application's ingress after the deployment controller finishes updating the deployments.
That said, if anyone REALLY wants to go the k8s route, it seems like starting with vanilla docker did allow me to get 75% of the work done before I needed to touch k8s itself :)
But, k8s offers a very good way to split the responsibility of managing the infrastructure and managing the applications that run on top. Many people work in medium to big corporations that have a bunch of people that are in charge of managing the compute infrastructure.
I certainly prefer k8s as an API between me and the infrastructure, as opposed to filing tickets and/or using some ad-hoc and often in-house automation that lets me deploy my stuff in a way that is acceptable for the "infrastructure guys".
However, if I use a managed k8s service, I probably don't have to think about any of that. I can focus on the metrics of my application, and not the cluster itself. At least, that's how I think it should work. I haven't used k8s in a while.
That's like saying running windows is the easy part, it's all the add-ons like Microsoft office that make it complicated.
It's running and is more hands off than without it. I'm using a managed digital ocean cluster. I no longer have to worry about patching the OS as it's all handled for me. I also don't have to worry about having a server with a bunch of specialized packages installed, although I suppose only using containers could have gotten me that far.
I haven't had a ton of headaches. So, I guess people's experiences may differ.
It's interesting to me that K8s always draws out the "you probably don't need it" comments.
To me this advice is only useful for tiny startups running a handful of web servers.
Disclaimer: I work at Google Cloud Run.
I run docker compose locally for development. For prod, I just use a different docker compose file (with some values changed, for example the postgres database url etc.). I do this from a 5 USD per month droplet/vm. I can launch multiple services like this for my microservices platform. I can use a hosted database solution for another 15 USD per month, to get backups etc. Also I get a generous 1 TB bandwidth and predictable performance for my system as a whole.
In the past I have used appengine and been bitten by their update times (took more than 20 mins for a single code update, things could have improved now). Also I need to write deployment artifacts for each service.
Now is there any benefit that cloud run (or any paas) could offer compared to this ? Would it be not easier to just stay with docker-compose and defer upgrading to kubernetes until you turn profitable or become unmanageable with a single VM ?
Cloud Run implements the Knative API, so you can actually take it away and run it on any Kubernetes cluster anywhere on a cloud or in your datacenter.
Source: at mid-sized company who has Istio in the stack.
What ends up happening is that your application consumes services (storage, analytics, etc.). You start using those services from the cloud provider, which makes sense as long as it is the right thing for your business (aligns w/ your cloud-native blueprint).
Kubernetes, by itself, is cloud-provider agnostic.
Overall though it is an amazing service and it is really fast and easy to use.
There is AWS Fargate for Kubernetes, AWS Elastic Kubernetes Service, DigitalOcean Kubernetes, Google Kubernetes Engine etc.
All of which are on the cloud and all of which don't require you to configure host OS etc. Some offer full control over node configuration e.g. EKS whilst others manage that for you e.g. Fargate.
I don't know which part you're not getting, but it appears that this person's intention is not to learn Kubernetes or deal with nodes in the first place.
Used nomad/consul/fabio at a previous job for running containers, it was very easy to adopt. Way less new concepts as well.
It's worth mentioning I later chose GCP managed kubernetes for a small cluster to run at my startup. I had to learn a few new things, but I'm not familiar of any "nomad as a service" offerings, so I went with k8s on Google cloud.
- remove the limit completely. Not a fan of this one since we really don't want a service going over a given limit...
- using the static management cpu policy  Not a fan because some services doesn't need a "whole" cpu to run...
Anyone has any other solutions? Thanks!
The symptom primarily seems to manifest that you get heavy throttling even though you haven't actually gotten to your cpu limit yet.
If you're just seeing heavy throttling AND its pegged at the limit, you haven't necessarily hit the issue and should raise the limit first and observe.
Also don't forget to eliminate other possibilities. We initially thought we were experiencing this issue and later discovered the app was just memory constrained and extremely heavy GC during certain operations was causing the throttling.
2. Same applies to dedicate cores for pretty much same reasons
Having said that if you really really want quota but don’t want shit tail latency I suggest setting cfs_quota_period to under 5ms via kubelet flag
This is not true at all. Autoscaling depends on CPU quotas. More importantly, if you want to keep your application running well without getting chatty neighbors or getting your containers redeployed around for no apparent reason, you need to cover all resources with quotas.
For such clusters, it's necessary to set up stuff like the LimitRanger (https://kubernetes.io/docs/concepts/policy/limit-range/) to put a hard constant bound between requests and limits.
In the meantime you can monitor rate of throttling and rate of CPU usage to limit ratio. Nothing stops you from doing this while also monitoring response latency.
On the other hand CPU request DOES potentially leave unused CPU cycles on the table since it's a reservation on the node whether you're using it or not.
Again needs may vary.
You can overprovision limit.
This is easy to test for yourself.
I've just checked Kubernetes' docs and I have to say you are absolutely correct. Resource limits are used to enforce resource quotas, but not autoscaling.
Once petabytes of data are out there in your GCP or AWS environment, "portability" will be costly due to extortionistic pricing of egress bandwidth.
The portability argument boils down to saying you are not boxed in. If things get bad enough you can move. This is a big long-term advantage for businesses because it means you can correct mistakes or adjust to changing business conditions. That's what most people who run companies are really looking for.
Lots of big companies that operate extensively in AWS/Azure/GCP don't go anywhere near the managed services they offer, because they end up being a horror show in terms of scalability, functionality and troubleshootability. Depending on your risk appetite, running Kafka on Fargate/EC2 is a lot more attractive than using Kinesis (for example).
At the petabyte scale there will probably be huge cost savings to using your own private cloud on your own hardware.
And it's not mutually exclusive. I've seen companies working very cost efficiently by using dedicated hardware for 99% of the workload and AWS or GCP for any experiments where they wanted to try out new features.
But as with any technology Guix and Nix are still decade ahead of the present and may pick up later when technology converge back to running application servers and other dependent software in isolation with user level namespaces.
Kubernetes try to solve one problem and create 10 other infrastructure problems to manage and instead of working on application, tie the company to a specific distribution or cloud service provider. So far there is nothing revolutionary in it unless the startup or company adopting k8s is google size operations.
From a software developer perspective which is the main audience of HN it will be popular as most of them dream or wants to work for company of size google. Startup founders want to solve the scaling problem like google in the beginning as everyone dreams to be google size from day one. Kubernetes complexity is useful at large scale for majority i.e. over 90% of the deployments simple containers, bare metal or VM with traditional configuration management will be sufficient.
I get the feeling people glance over the documentation and assume that you MUST use every single feature. Most of the features are only there when you need them for supporting advanced use.
Where does Nix/Guix manage dynamic binpacking of applications to servers (preferably without a blood sacrifice on the altar of "infinite recursin")?
Where do they handle nicely (de)provisioning and attachment/detachment of special resources, whether those are filesystems, block devices, any kind of special device etc.?
How about integrating various special networking resources like load balancers?
No, NixOps and related do not handle it well, as they need static assignment upfront and can't dynamically adjust as the system runs. Even at my small scale, all those features above are crucial, and k8s let's me not worry about problems most of the time (Rarely we have a black swan, that pretty much always ends up being "not k8s fault that network died or nginx got hung")
There are probably 10x more engineers that have experience with k8s than with guix though so ... perhaps that could be a factor though not a reason to change out the infra completely.
Nix is great to build the image and manage changes to dependencies.
Start here: https://kubernetes.io/docs/tutorials/ Look at all the diagrams in the tutorials but don't bother with the interactive stuff.
Then learn the more detailed concepts here: https://kubernetes.io/docs/concepts/
Once you understand the lingo and the general idea of what the different functions are supposed to do, just copy some example deployments and try to get something of your own working.
Don’t do what I did: googling and going through random top hits. Most of these are 1-off blog articles that revolve around “install Helm and then do these 6 things” or “just kubectl apply random.link.com/some-script”.
Doing that just leads to tons of confusion and anger.
My recommendation: Suck it up and read through the official kubernetes docs. Their docs aren’t written in a way to easily explain core concepts unfortunately. However, slogging through it will give you some initial exposure to concepts and will let you know where to go back to later when your making mental connections.
Next, look for k8s tutorials from Digital Ocean and Linode. In my opinion, they’re the best written guides for demonstrating how to get from A to B.
You’ll start running through those guides and be referencing back to the official docs. Gradually bridges will form on your head and you’ll get an intermediate, functional level of competence.
If you grok all that, then you are ready to grok the K8s Container Runtime Interface. And, if you really grokked it, you understand that Docker is besides the point, isolation and namespaces are optional, and so forth. All k8s demands from the "container runtime" is that a thing has a name and a defined lifecycle.
Once you understand how k8s works at the pod and node level then it should be perfectly obvious how it works at higher levels. That's why I really want to see someone write the bottom-up guide!
Now I’m also more leaning towards the official docs as a recommendation, because they should always be more up to date... nevertheless, “Kubernetes: Up and Running” took my fear off this (at first) complex architecture. In the end, K8s is not that difficult to understand and the involved building blocks make sense, after you get the hang of it.
By the way, Microsoft is giving away “Kubernetes: Up and Running” second edition for free currently: https://azure.microsoft.com/en-us/resources/kubernetes-up-an...
EDIT: Upon further investigation, I see that there's not much technical content provided here. So it goes. Thanks anyways.
This is exactly the kind of word salad approach to explaining k8s that I complained about in my other comment. All of the k8s tutorials are written by (or recorded by) people with no idea what they are talking about.
There are plenty of great developer distributions out there (k3s, kind, minikube, microk8s), but those are single node only, and aren't meant for production use.
I'm still searching for a solid guide on how to get k8s installed on your own hardware. Any suggestions would be very appreciated!
I've found this pretty good starting point. Keep in mind the book is still a work in progress. Note, I haven't actually run k8s in production, but this book has helped me get something up and running in VM's pretty quickly using Ansible on top of kubeadm.
Keights and Kubespray both use Ansible, however they do it in a very different way. (Disclaimer: I haven't used kubespray, only looked over the documentation). Keights uses Ansible roles to build CloudFormation stacks to produce a cluster. The nodes in the cluster bootstrap themselves using systemd services that are baked into the AMI; Ansible does not run on the nodes in the cluster. Kubespray, as I understand it, uses a traditional Ansible approach of pushing configurations over ssh to nodes in its inventory. To my knowledge, it does not actually build the machines in the cluster, it just configures existing machines. Keights does the full end-to-end automation to bring up a working cluster, including the creation of all required AWS resources (autoscaling groups, a load balancer for the apiserver, security groups, etc - though you do provide certain resources as parameters, for example your VPC ID and subnet IDs, due to the aforementioned requirements to fit into locked-down environments).