
A Gentle introduction to Kubernetes with more than just the basics - feross
https://github.com/eon01/kubernetes-workshop
======
Apocalypse_666
After having just spend most of the day yesterday trying to nurse a failing
Kubernetes cluster back to health (taking down all of our production websites
in the process), I’ve come to completely loathe it. It felt more like
practicing medicine than engineering: try different actions to suppress
symptoms, instead of figuring out a root cause to actually prevent this from
happening again. While it is a pretty sweet system if it does run, I would
strongly advice against anyone trying to manage their own cluster, as it is
simply to complex to debug on your own, and there is preciously little
information out there to help you

~~~
Apocalypse_666
I wrote this whole thing below as a reply to someone stating that I should
just stop complaining and figure it out, but that comment has since been
deleted. Figured sharing my frustration might be cathartic anyways :)

If only I could!! That’s exactly the frustrating part: there seems to be no
way of grokking what goes on under the hood, and there are so many different
ways of setting up a cluster and very few have any information about them
online whatsoever.

As a practical example, what happened yesterday was that all of a sudden my
pods could no longer resolve DNS lookups (took a while to figure out that that
was what was going on, no fun when all your sites are down and customers are
on the phone). Logging into the nodes, we found out about half of them had
iptables disabled (but still worked somehow?). You try to figure out what’s
going on, but there’s about 12 containers running in tandem to enable
networking in the first place (what’s Calico again? KubeDNS? CoreDNS? I set it
up a year ago, can’t remember now...) and no avail in Googling, because your
setup is unique and nobody else was harebrained enough to set up their own
cluster and blog about it. Commence the random sequence of commands I’ll never
remember until by some miracle things seems to fix themselves. Now it’s just
waiting for this to happen again, and being not one step closer to fixing it

~~~
antocv
Well that sounds like a problem of "I run my own cloud". And not a problem of
Kubernetes. You dont remember how you set it up, oh well.

If you use a managed kubernetes (not in aws since they suck, eks is not really
managed). Like gke or aks, then you skip the whole "there is a problem in my
own cloud of my own making".

btw, I also encountered DNS problems in kubernetes, on ACS, it took 5-10
minutes to resolve, and was caused by ACS not having services enabled to
restart dns upon reboot, lol.

~~~
dijit
The whole promise of Kubernetes (and, coincidentally, containers) is that you
are not locked to a platform/provider/OS.

Reading this comment made me realise that often new technology is adopted
because it is optional and promises options. But those options quickly shrink
away and suddenly you’re locked into it.

Not to invoke a controversial name. But this is what happened with systemd.

~~~
GauntletWizard
Yes! You are not locked to a platform/provider/OS. Your GKE cluster operates
more or less like your EKS cluster operates more or less like your cluster in
Azure, DigitalOcean, etc. Kubernetes is a _deployment platform_ you target and
complaining that the things two layers under the hood of that are different is
a false analogy.

Moving from one Kubernetes Provider to another is not zero time. You need to
learn some differences in the way GKE ingresses vs AWS ELBs work, etc. It is a
substantially more tractable problem than the differences between Cloud
Bigtable and DynamoDB, and that one is still a tractable problem.

The way to fight lock-in is is not, and has never been, "These two providers
offer exactly the same service". It has been about avoiding "These two
providers offer nothing that is analogous, and their documentation is directly
written to encourage using practices that do not port". It has never been an
all-or-nothing thing.

------
antocv
This is yet another one of those "introductory" articles.

The whole field it seems everywhere is filled with "introductory" "gentle"
books/articles, and then "this is how you do reusable rocket science with x".

Pro tip: to understand kubernetes in between, go read the manual pages of
Linux networking and get a really good grip on iptables. Go read the manual
pages for linux namespaces, cgroups and containers with lxc.

Why dont people get the basics of the parts of the tool they are going to use,
first, instead of trying to "understand a tool"? You wont succeed doing
anything with kubernetes, if you come from say macosx or windows envrionment,
and have no clue how/what kubernetes is built on.

~~~
StreamBright
Sure, I know iptables, namespaces, cgroups and containers and lxc. I stopped
right here and did not bother to learn Kubernetes because all of this
knowledge gives me enough to run anything I want, websites, applications,
hadoop clusters and so on without it. I can debug my stack and resolve any
issues with confidence, have upgrades and do autoscaling (using AWS). Why
should I even think about Kubernetes? Now the real question is, if you need to
learn all these to run Kubernetes than is it really tool they are trying to
sell it is? One of the promise I often hear is that you don’t need systems
knowledge or SREs to run it. Yet I read these stories and many customers reach
out for help because they have an undebugable problem in production and nobody
can help. I learned in the last 20 years working as a systems engineer that
the best tools or stacks are ones easy to understand, debug and fix. Black box
computing is the worst, when things happen for no good reason and it is hard
to find out the root cause.

~~~
MrGman
How is AWS not black box computing? Isn't it worse than a black box? Because
it's a black box that's sitting in a black building that you don't have access
too?

Disclaimer: I'm a n00b when it comes to web stuff.

~~~
StreamBright
You know when it breaks (built in monitoring) and there are SLAs. If it is
broken Amazon will fix it. I can work around it, using caching proxies for S3
for example. There are other approaches to be more resilient against cloud
outages. The most important difference between self hosted Kubernetes vs cloud
native services is that who will fix it.

------
ggregoire
I'm wondering if Kubernetes would be the right choice in our use case. Or if
something like Ansible would be better suited and easier to setup and use?

I'll soon have to manage 50 remote bare-metal servers, all of them in
different cities. Each one of them will run the same dockerized application.
It's not 1 application distributed in 50 servers; it's 50 instances of the
same application running in 50 servers, independently.

A frequent use case is to deploy a new version of this application on every
server (nowadays I do it manually, it's OK since I manage only like 10
servers).

A nice-to-have would be to avoid downtimes also, when I update the application
(nowadays I have a downtime of 2-5 minutes (when it goes well), which matters
for us).

~~~
rochacon
No, as @dragonsh said, Kubernetes will be a bad choice for you. Kubernetes is
complex, specially its network stack, you'll need to setup an VPN from each
site to your control plane site to keep some basic functionality working
(kubectl logs, for example).

If you don't care about a centralized API to probe status and manage each
instance, Ansible should be enough to orchestrate these installations and with
little effort (that also depends on the application at hand) getting zero-
downtime rollouts with Docker can be easily done with it.

However, if you want a single control place to probe status and want to avoid
writing your own rollout scripts, Hashicorp's Nomad [1] might be a good
solution for this. It is a lot simpler than Kubernetes while still giving you
nice primitives to describe jobs/services, health checks, rollouts strategies
and etc. Treat every site as a datacenter of its own, setup a job of type
"system" (akin to Kubernetes DaemonSets) and all you need on these sites is
internet access to your HTTPS endpoint of the Nomad control plane.

If you want to talk more about this, hit me up on Twitter or Telegram, I use
@rochacon as my handle virtually everywhere.

[1] [https://www.nomadproject.io](https://www.nomadproject.io)

Edit: grammar and typos

~~~
MuffinFlavored
How can zero downtime deploys be done with Docker?

I had to write my own custom blue/green deploy script to hot reload traffic to
proxy_pass definitions in nginx upstream configs since I don’t use Docker.

~~~
jeremija
IIRC, if you use docker swarm it wil handle the routing for you. So if an app
is running on port 8080 on two swarm nodes, if you access port 8080 on each
node, you might actually be accessing another node.

Since docker swarm knows if an instance is down, it will know to not use that
instance.

See
[https://docs.docker.com/engine/swarm/ingress/](https://docs.docker.com/engine/swarm/ingress/)

~~~
Axsuul
I run on Docker Swarm and there are hiccups as one container goes up and
another goes down during deployment.

------
skboosh
A major problem I've seen using Kubernetes is it's difficult to bring up an
entire cluster from scratch, play with it and tear it down. So I wrote
Sugarkube [1] which lets you launch an ephemeral cluster, install all your
applications into it (including creating any dependent cloud infrastructure)
and tear it down again. This means each dev can have their own isolated K8s
cluster that's in parity with production. And you can release the same code
through different environments. In fact your prod environments can become
ephemeral too - instead of doing complicated in-place upgrades you could spin
up a sister cluster, direct traffic to it and then tear down the original. Or
you could spin up a cluster to test the upgrade before repeating it in your
live environment.

Based on my own experience I believe ephemeral clusters can solve a huge
number of problems dev teams face using Kubernetes.

Sugarkube currently supports launching clusters using Minikube, EKS and Kops
and we'll be adding provisioners for GKE and Azure in future. Sugarkube also
works with existing clusters, so you can use it deploy your applications. It's
a sane way of managing how to ship infrastrucuture code (e.g. Terraform
configs) with your applications that need them.

Sugarkube also supports parameterising applications differently per
environment - you could almost view it as something like Ansible but that was
written with Kubernetes in mind. And it's in Go so it's way faster than
Ansible (an early POC was actually written in Ansible and it was very slow).

I've just finished intro tutorials for deploying Wordpress to Minikube [2] and
EKS [3]. I'd be keen to hear feedback. We just tagged our first proper release
earlier this week and it's ready to try now.

[1] [https://docs.sugarkube.io/introduction/ephemeral-
clusters/](https://docs.sugarkube.io/introduction/ephemeral-clusters/)

[2] [https://docs.sugarkube.io/getting-started/tutorials/local-
we...](https://docs.sugarkube.io/getting-started/tutorials/local-web/)

[3] [https://docs.sugarkube.io/getting-started/tutorials/dev-
web/](https://docs.sugarkube.io/getting-started/tutorials/dev-web/)

~~~
vitalus
I definitely agree with you - my team switched to an "ephemeral cluster" model
which allows us to very quickly spin up an entirely new cluster and drain
traffic to it as needed.

It's something that we've ended up implementing on our own with a lot of
Terraform, but that's had its own obstacles and is something of a small
maintenance burden. I'll be taking a look at sugarkube!

~~~
nhoughto
Not quite the same, but good for CI jobs

[https://github.com/bsycorp/kind](https://github.com/bsycorp/kind)

~~~
swsieber
Oh my - there are two distinct k8s bootstrapping projects named "kind" :(

[https://github.com/kubernetes-sigs/kind/](https://github.com/kubernetes-
sigs/kind/)

------
vasco
There are advantages to using minikube for local development if you are
interested in replicating and testing the whole flow. I think these are most
important for small teams of 1-5 people where everyone needs to understand the
whole infrastructure stack and regularly performs changes down to the
kubernetes resources. That said, I've found that for bigger teams where you
have a few people concerned with keeping k8s stable, and most people
developing the actual application logic, docker-compose is still the most no-
frills-easy-to-understand option for local development.

------
kn100
I've seen a deployment of a large number of ec2 instances + other Amazon
services handled with Puppet+Ansible, and I've also seen a somewhat smaller
but still fairly sizeable deployment using a custom Kube cluster. The custom
Kube cluster was cool and definitely seemed to make those working with it
happy, but I was much happier working with the Puppet+Ansible bare metal
setup. It's just easier to figure out what's wrong. I am now working with
hosted Kubernetes, which seems to be much less of a mental headache.

------
peterwwillis
Is there a really abbreviated reference to all of core kubernetes? Like, one
page with just all the categories, services, objects, attributes, schemas,
etc, for every component. I know it would be 1000 pages long, but it would
allow quickly skimming as a reference, as well as getting a very quick low-
level glance at _everything_ involved.

Guides like OP's are great to get started, but I want to know that
weird_flag_that_almost_nobody_ever_sets exists in random_service_parameter.

~~~
rochacon
Something like this?
[https://kubernetes.io/docs/reference/generated/kubernetes-
ap...](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/)

~~~
Yen
+1 for the reference documentation.

I've often been able to write complex and correct manifests, just by
recursively following the links and filling in the YAML as appropriate.

~~~
blondin
i mean reference documentation is nice if you understand the core concepts. my
struggle is with grasping those core concepts and how they all fit together to
solve the orchestration problem.

for now, a few books (kubernetes patterns is the current one i am reading) and
articles are helpful. i would like to note that some of these concepts had
their names changed overtime. i suppose there will be more changes as the
platform stabilizes. mind you guys that kubernetes is only 5 years old...

------
mfbx9da4
I hate the fact that most of my time is spent configuring rather than
building.

------
MichaelMoser123
I have a short text on kubernetes operators.
[https://github.com/MoserMichael/cstuff/releases/download/kbo...](https://github.com/MoserMichael/cstuff/releases/download/kbook-
upd/kubernetes-book.pdf) learning this stuff got me something of an overview
of what is going on there.

------
domenukk
Tried to set up a Kubernetes cluster with IPv6 the other day. These hipster
tools don't care about IPv6 :(

------
th3iedkid
Is mesos dead ?

~~~
nih0
no, It's very much alive, there's just no hype around it anymore

------
auslander
A generic engineering rule: fight complexity, a.k.a KISS principle. It takes
additional effort and courage. You'd need to fight for that project time too.
You'll live longer (and your systems too), but probably on your next job :)

