One thing I've found about k8s is running a cluster can be very costly, and using a managed cluster on one of the big three clouds is even more so. Also, when rolling your own cluster it requires a lot of elbow grease to get things in a working state, and even more to keep things working.
It feels awesome from a developer standpoint once you work out all the kinks. It's very natural describing your service with a bit of yaml, and then being able to then spawn it into your infrastructure and have it automatically scale out.
If there was a cost effective k8s cluster you could rent then that would be the thing that finally kills traditional infrastructure for me.
That's the exact same feeling I get too. If I'm a mom/pop shop online, why can't I get a k8s cluster w/just 1 machine for $5/month? I mean the big three (4 w DO) can all get you a bare machine for $5/mo that can run Docker. Why not give the k8s management side for free for those people?
You can get a small cluster from Digital Ocean pretty cheaply. It's more than $5 (I forget the exact price) but they'll provide a free master node and then you just pay for one additional kubelet (and you can add more as you need them). I have used it for one hobby project and have been pretty happy with it.
Same with Linode, the management machines are not massive scale HA control planes like you get with the expensive hosts (It seems like Linode may run a single 2GB droplet), but for most of my clusters I don't need it.
IaaS model is more 'everything is chargeable in small units', a free control plane doesn't really gel with that IMO, and also they have some huge users that it'd cost a lot to run the control plane for, pushing up the price of what you connect to it (or something must give) and making more PaaS-style like DO & Scaleway anyway.
From my experience, deploying to a k8s cluster managed by someone else is extremely simple and straightforward. Tell it which container to run and off it goes. You get zero-downtime rolling deploys, automatic restarts on crashes, easy rollbacks, service discovery (if you have multiple services) and more for free.
Managing a cluster is the tricky part of k8s but just running your stuff on an existing cluster is extremely straightforward. I could definitely see the attraction of a cheap managed k8s cluster with a single node. Maybe hosted k3s or something.
A ton of the value from k8s is in the programmatic and replicable configuration. 1 computer pets are still bad in a mom and pop shop. 1 computer livestock is still better.
In this case the state can come from a centrally managed k8s controller. Something that most cloud providers provide and have good SLAs on.
Kubernetes just makes that story so much simpler. Heck a good cloud provider can just handle any machines fucking up automatically without any human intervention on your side.
But being able to blow away a k8s node and bring it back up in 2 minutes (or have k8s auto-spawn a replacement node) is WAY faster than having to reinstall wordpress, reinstall nginx, set IP addresses, fiddle with your DNS, restore a database from a backup, etc, etc, etc.
Not to mention everything you have to do to get a machine to parity with Kubernetes--set up SSH, process management, monitoring, logging, etc. I don't think I'd go so far as to say Kubernetes is the right tool for this use case, but I certainly empathize with the sentiment.
Well, in this case the pop is me, and I know k8s but I don't want to invest 15-35 / month on a side project but would still rather have a decently managed site rather. I don't want to go back to FTP crap at this point.
A car is a hugely complicated machine and I'm sure in the 1910s people said only some people will ever need them. Bad analogy considering cars kill lots of people but take that part out :)
And, not everything should be in a container. Just because you can does not necessarily mean you should. Containers are awesome but they are not a cure-all silver bullet.
What is your utilization? IMHO I would rather do one EC2 instance then have EC2 Batch spot and lambda workers take the other load. In a good design most media assets should never hit the full time provisioned server.
It feels awesome from a developer standpoint once you work out all the kinks. It's very natural describing your service with a bit of yaml, and then being able to then spawn it into your infrastructure and have it automatically scale out.
If there was a cost effective k8s cluster you could rent then that would be the thing that finally kills traditional infrastructure for me.