
The Kubernetes Effect - sdiepend
https://www.infoq.com/articles/kubernetes-effect
======
archgrove
I'll stand by my assertion that for 99% of users (maybe even 99.99%),
Kubernetes offers entirely the wrong abstraction. They don't want to run a
container, they want to run an application (Node, Go, Ruby, Python, Java,
whatever). The prevailing mythology is you should "containerize" everything
and give it to a container orchestrator to run, but why? They had one problem,
"Run an app". Now they have two, "Run a container that runs an app" _and_
"maintain a container". Just give the app to a PAAS, and go home early.

Most startups - most large companies - would be far better served with a real
PAAS, rather than container orchestration. My encounters with container
orchestrators is that ops teams spent inordinate amounts of time trying to
bend them _into_ a PAAS, rather than just starting with one. This is why I
don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they
solve entirely different problems. My advice to almost every startup I speak
to is "Just use Heroku; solve your business problems first".

The article also mentions it enables "new set of distributed primitives and
runtime for creating distributed systems that spread across multiple processes
and nodes". I'll throw out my other assertion, which I always though was
axiomatic - you want your system to be the _least_ distributed you can make it
at all times. Distributed systems are harder to reason about, harder to write,
and harder to maintain. They fail in strange ways, and are so hard to get
right, I'd bet I can find a hidden problem in yours within an hour of starting
code review. Most teams running a non-trivial distributed system are coasting
on luck rather than skill. This is not a reflection on them - just an inherent
problem with building distributed logic.

Computers are fast, and you are not Google. I've helped run multiple thousand
TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a
few services. I'm now helping a startup unpick it's 18 "service" containerised
system back to something that can actually be maintained.

TLDR; containers as production app deployment artefacts have, in the medium
and long term, caused more problems than they've solved for almost every case
I've seen.

~~~
brightball
I tend to agree with you and it's one of the biggest reasons that I'm a fan of
Elixir.

Here's the path that leads to K8s too early.

1\. We think we need microservices

2\. Look how much it will cost of we run ALL OF THESE microservices on Heroku

3\. We should run it ourselves, let's use K8s

One of the big "Elixir" perks is that it bypasses this conversation and lets
you run a collection of small applications under a single monolith within the
same runtime...efficiently. So you can built smaller services...like a
monolith...with separate dependency trees...without needing to run a cluster
of multiple nodes...and still just deploy to Heroku (or Gigalixir).

Removes a lot of over-architectural hand-wringing so you can focus on getting
your business problem out the door but will still allow you to separate things
early enough that you don't have to worry about long term code-entanglement.
And when you "need" to scale, clustering is already built in without needing
to create API frontends for each application.

It solves a combination of so many short term and long term issues at the same
time.

~~~
lilactown
100% agreed. A lot of the cloud computing products are simply re-
implementations of what was created in the Erlang/BEAM platform, but more
mainstream languages. IMO it's cheaper to invest in learning Erlang or Elixir
than investing in AWS/K8s/etc.

Elixir and Erlang are basically a DSL for building distributed systems. It
doesn't remove all of the complications of that task, but gives you excellent,
battle tested, and non-proprietary tools to solve them.

~~~
pjmlp
And JEE :)

------
falcolas
<rant>

And yet finding people who can reliably install K8s from scratch, who
understand what's going on under the hood, remains remarkably close to 0.

How many people can, within a few hours, tell you how Kubernetes runs DNS, and
how it routes packets between containers by default? How do you run an
integrated DNS which uses, say, my_service.service.my_namespace instead of
my_service.my_namespace?

I've found that most installs of k8s have been made using defaults, using
tooling that Google has provided. We hired one such administrator, but when
asked anything outside of how to run kubectl, they just shrugged and said "it
never came up".

The codebase is vast, complicated, and there are few experts who live outside
of Google. And it's getting more vast, more complicated on a quarterly basis.

It bothers me how far operations has gone from "providing reliable systems on
which we run software" to "offload work onto the developer at any cost".

</rant>

I realize that a lot of this is because of scarcity. The good devops folks
(i.e. those who are both competent generalist sysadmins and competent
generalist programmers) are few and expensive. That makes pre-packaged "full
stack" solutions like GAE, Kubernetes, and Fargate very appealing to
leadership.

"You don't need an operations department to act as a huge drain on your
revenue, just re-use your developers" holds a lot of appeal for those high up
in the food chain. It's even initially appealing to developers! But in the
end, it makes as much sense as re-using your developers to do customer
service.

~~~
geggam
My question is this. Why does the container world use NAT.. ( 3 layers to get
out of container to base host in k8s ) ... and not use routing ?

Is it just the container devs dont know routing ?

~~~
puzzle
Kubernetes is the opposite. NAT is explicitly not required:

[https://kubernetes.io/docs/concepts/cluster-
administration/n...](https://kubernetes.io/docs/concepts/cluster-
administration/networking/#kubernetes-model)

E.g. on AWS you might have all of a node's pod IPs on a bridge interface, then
you talk to pods on other nodes thanks to VPC route table entries that the AWS
cloud provider manages. NAT happens only when talking to the outside world or
for traffic to Amazon DNS servers, which don't like source IP addresses other
than those from the subnet they live in.

~~~
falcolas
My memory is a touch fuzzy, but to route traffic out of a container in AWS,
you have to either NAT thorough the instances network adapter, or attach an
ENI to the container. However, you only get one ENI per vCPU in a VM (at least
until Amazon finishes its custom NICs). What I'm really fuzzy on is whether
the instance itself consumes one of those allocated ENIs.

That is, if you're running off a m4.2xLarge instance, you get a maximum of 8
ENIs - 8 containers if you want to use only VPC routing. For some services,
this may be OK, but for many others (most?), it's far too few.

~~~
puzzle
What's the destination? If it's the outside world, yes, you need NAT for state
tracking and address rewriting, since the rest of the AWS infrastructure knows
nothing about the pod CIDR (I guess you could set up a subnet for it and run a
GW there).

For pod to pod, if you're OK with the limitations of 50 routes per VPC route
table (you can open a ticket to bump that to 100, at the cost of some
unspecified performance penalty), then you don't need NAT.

Otherwise, you can use something like Lyft's plugin, which does roughly what
you describe. On a m4.2xlarge you only get 4 ENIs, but each of them can have
15 IPv4 and 15 IPv6 addresses, which the plugin manages. They assign the
default ENI to the control plane (Kubelet and DaemonSets), so you should get
45 pods.

------
sacheendra
A consequence of the "Kubernetes Effect" is that while distributed systems are
easy to build and use, a lot of developers lose sight of the fundamental
problems which make distributed systems difficult.

For example, the sidecar in a sidecar pattern might fail while the application
is running and the system can get stuck in weird states. The developer still
needs to understand fundamentally how the system works.

Eschewing deeper knowledge just because it is easy to use is trap in this
case. While the article compares Kubernetes to JVM, Kubernetes can fail in a
lot more hard to debug ways than the JVM right now. I don't know if this
semantic gap between distributed systems like Kubernetes and monolithic
systems like JVM can ever be bridged.

~~~
majidazimi
> A consequence of the "Kubernetes Effect" is that while distributed systems
> are easy to build and use, a lot of developers lose sight of the fundamental
> problems which make distributed systems difficult.

I would extend this to cloud as well. The more prevalent cloud becomes, the
more ignorant developers become. It's like: I have Mathematica license, who
cares how to calculate function derivative?

~~~
RRRA
More generally, society's achievement currently rely on a workforce that gets
more and more specialized.

We are bound to fragment every sector into sub-niche where specialists in
functions, general programming and infrastructure resources cooperate on their
boundaries without being able to quite understand what the others are doing.

------
thedevopsguy
Kubernetes by itself may be daunting for most teams.

But I'm not sure I understand the backlash. Once you've built your application
and it's been packaged (containerized) and deployed why would anyone care how
its run. Also running a container in production and orchestration seem to be
conflated somewhat in this thread and the use cases are very different.

You can think of Kubernetes as an Automated SysAdmin . This is a bit reductive
I know but it is useful to think of this way. You ask the sysadmin to run
something for you and they tell you how to package it (tgz, war, zip etc) and
they run it for you on hardware.

The level of engagement that a dev has with getting his app running on
hardware is no different to that of dealing with a sysadmin and with the admin
requesting that your app is packagedin a container.

Kubernetes out of the box will give you most of this functionality as long as
you keep state outside of the cluster. There are also options on how to make
the experience smoother. There also these tools to help too:

* Openshift * Kubernetes + Rancher * Mesos

If you need orchestration and scheduling. I am a little perplexed.

------
mixmastamyk
Any one have opinions on the Kubernetes admin cert linked? Wonder if it would
significantly help in getting work and skipping the interview BS?

[https://www.cncf.io/certification/expert/](https://www.cncf.io/certification/expert/)

~~~
markbnj
I've been working with kubernetes since 2015 and running production workloads
on GKE since 2016. Since you asked for opinions, mine is that the certs don't
matter very much. CNCF plays them up, and they will probably have some impact
in larger orgs as enterprises get on the train, but in the open source
community from which most of the kubernetes momentum emanates there has never
really been a ton of respect for formal certification programs, and this
doesn't feel any different to me.

~~~
dsnuh
Just wanted to chime in to generally second this opinion, but with one
exeception as it relates to hiring.

While I don't think having a cert would help you get hired necessarily, it
would probably influence a decision to get an interview. What really matters
is if you know how to do real-world operational tasks with the knowledge,
which will show up if your technical interviewers know k8s. If you are the
first person they are hiring at the company to begin their k8s project, then
you might have a real advantage with a cert.

Personally, I've never been one to give undue respect to many of the certs on
the basis of having them alone, but it can depend on where you interview. Some
places love certs.

~~~
mixmastamyk
Yep, I suspected as much. Was just jealous of some nurses I know where their
job interviews consist of:

Is your license up to date? Are you an ax-murderer? Can you start Monday?

~~~
markbnj
My wife has been a nurse for 12 years and there's some truth to that, but
"license" in their case goes way beyond kubernetes certification :). It's like
if someone hands you an up-to-date pilot's license and log book you know they
can fly proficiently. We don't have that in software, as far as I know, so
that automatic respect for the document isn't there.

~~~
dsnuh
Yes, and you also don't have nurses having to train for different "flavors" of
humans (I'm speaking in a general practice sort of way, of course there are
speciality nurses). If you know how to draw blood, you don't have to learn a
whole new system based on the type of hypodermic needle you use. In the
Kubernetes world, you'd need to know how to use AWS, GKE, your own custom
stack, etc, in addition to knowing how to draw blood on this particular
version of this flavor of human.

