I disagree. Maybe it's because we haven't gone "full Kubernetes" where every cronjob is now a Kubernetes cronjob, or what have you, but for all the complexity we've had to overcome, we've seen substantial benefits.
For example, autoscaling instances of my data pipeline apps based on Kafka lag.
Was it complex? Yes. We had to expose Kafka topic lag as a metric to Prometheus, then configure the k8s metrics server that HPAs (Horizontal Pod Autoscalers) use to scale to pull that in from Prometheus using k8s-prometheus-adapter as a custom metric, then set reasonable scaling limits in the data pipeline apps HPAs.
Was it worth it? Fuck yes. We no longer have to worry about data arriving out of time at our data warehouse, because the scaling we've configured (in YAML, god rest its dirty soul) ensures that our data will always arrive at the data warehouse within 5 minutes, which is ideal for a near real time reporting product that greatly enables our end users.
Is Kubernetes worth it? For us, definitely. For anyone else? Really depends on your use case, don't cargo cult this shit.
Where do you work? I'd like to pitch my radical idea of edge consolidated cloud computing.
Yep, we could handle it easily on one machine, if we gave it unfettered access to all the resource, but our throughput varies during the time of day, so dedicating a machine to it to handle the peak throughput wouldn't make financial sense at 3am at night. So it's far simpler to scale pods with known resource usage as needed.
It also helps prevent issues when throughput suddenly doubles because of a business decision that you're left out of the loop on.
So autoscaling pods is ideal for our use case.
Is your aws bill really under $500 a month?
Will see if history repeats itself and we will see be imperative systems.