
Everything I know about Kubernetes I learned from a cluster of Raspberry Pis - alexellisuk
https://www.jeffgeerling.com/blog/2019/everything-i-know-about-kubernetes-i-learned-cluster-raspberry-pis
======
raesene9
A good method for learning how clusters work and playing with them without
having to spend a lot of time re-building when you break things is kind
([https://kind.sigs.k8s.io/](https://kind.sigs.k8s.io/)).

Each node is a Docker container, but the version of Kubernetes running inside
it is vanilla Kubeadm, so it's quite representative of what "real" clusters
would look like.

The great thing about it is you can spin up a cluster in < 2 mins locally to
try things out, then it's a single command to delete.

~~~
mkesper
How does this compare to k3s?
[https://rancher.com/docs/k3s/latest/en/](https://rancher.com/docs/k3s/latest/en/)

~~~
bndw
Kind is explicitly not for production workloads[0]:

    
    
      Non-Goals: Being “production workload ready” - kind is meant to be used:
    
        for testing Kubernetes itself
        for testing against Kubernetes (EG in CI on Travis, Circle, etc.)
        for “local” clusters on developer machines
        NOT to host workloads serving user traffic etc.
    
    

K3s seems happy to support production workloads. It's a single binary and gets
you a cluster in one command:

    
    
      curl -sfL https://get.k3s.io | sh -
      sudo kubectl get nodes
    
    

Personally, I use both Kind and k3s. Kind is great for running locally and k3s
is awesome for single-node kubernetes on a vps.

[0] -
[https://kind.sigs.k8s.io/docs/contributing/1.0-roadmap/#non-...](https://kind.sigs.k8s.io/docs/contributing/1.0-roadmap/#non-
goals)

~~~
lhuser123
What are the benefits you find in using Kubernetes for a single-node cluster
on a vps?

~~~
bndw
I host a few personal websites and Kubernetes allows me abstract the server
setup. I don't need a highly available cluster and found k3s on a single node
to be a nice replacement for manually configuring a reverse proxy, systemd
services, deployment scripts, etc.

------
ricardbejarano
Everything I know about Kubernetes I learned from a single-node
(pseudo)clustered refurbished ThinkPad X201.

It cost me 90€ at an IBM refurbished sell. It is downstairs by the router, and
it has been hosting everything for me except my email and my blog (which I
want to host but I'm not sure about the reliability of my ISP's service, this
morning it went down for 5h12m, without prior notice or anything).

It is amazing how much you learn from doing stuff. I'm currently on my 3rd
year of univerity for CS, so I've tried the academia-style learning process,
reading books on my own and doing stuff on my own. The latter is the best
method by far.

~~~
mpfundstein
I only really learn by doing stuff. I read books usually to get started but
mostly quickly have to turn to building something. Then revisiting the book on
the go. If I only read the book/paper/whatever, I usually ‘think’ I get it but
nearly always that isn’t the case :-)

------
vyshane
3 years ago, I also wanted a bare metal cluster for my homelab. I wanted
x86-64, low power consumption, small footprint, and low cost. I ended up
building this 5 node nano ITX tower:

[https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-
to...](https://vyshane.com/2016/12/19/5-node-nano-itx-kubernetes-tower/)

I think that the exposed boards adds to its charm. Doesn't help with dust
though.

~~~
geerlingguy
Very nice; I've considered doing something similar and running some production
sites on it from my home, but the limitation has always been my terrible
Internet bandwidth through Spectrum.

We almost got Verizon gigabit fiber a few years ago... then AT&T ran fiber to
the front of my neighborhood last year, and then never ran it to the houses.
As it is, I'm stuck with 10 mbps uplink, which is not enough to be able to do
most of what I would want to do with a more powerful local cluster.

------
pstadler
Great article! Never stop tinkering.

Here‘s how I got to know Kubernetes:

By end of 2016, the Kubernetes hype was just about to pick up real steam. As
somebody who always liked the idea of running something like an own cluster in
the cloud, I attended KubeCon Europe in early 2017. The event was sold out,
and took place in a venue almost too small for the number of attendees. It was
great. During the event I was just about to finish the Hobby Kube[1] project.
Back then weren’t any guides that addressed all the problems encountered when
running Kubernetes on a budget, using low cost VPS on random cloud providers.
So I dived into the subject in the second half of 2016 and started writing a
guide, including automated provisioning using Terraform. I discovered
WireGuard in the process of looking for a solution to efficiently securing
network traffic between hosts. This still makes me feel like I was an early
adopter to something that’s becoming hugely popular.

If somebody would like to add a Terraform module for deploying to Raspberry
Pi, please ping me or open a PR here[2].

[1] [https://github.com/hobby-kube/guide](https://github.com/hobby-kube/guide)
[2] [https://github.com/hobby-kube/provisioning](https://github.com/hobby-
kube/provisioning)

------
SirMonkey
I learned k8s with some NUCs we had laying around at work. Might be easier
than Pi, but not as cheap. Some things I used:
[https://metallb.universe.tf/](https://metallb.universe.tf/) (LoadBalancer)
[https://cilium.io/](https://cilium.io/) (Networking)
[https://rook.io/](https://rook.io/) (Persistent Storage)

~~~
bogomipz
Does Rook give you the equivalent of EBS root volumes for your nodes then? Is
that the function you have it providing? Does it offer something beyond using
local host storage and minio?

I ask because I've generally been confused about the use case for Rook despite
having read the "what is Rook?" paragraph many times on the project home page.
My assumption is that it lets you build your own internal cloud provider. Is
that correct?

~~~
moondev
Not directly, rook provides a storage backend for in-cluster workload
persistent volumes.

In theory you could manage kvm machines with kubevirt and back the machines
with PVCs from rook. I have not tried this and would be curious how the
performance is.

~~~
ownagefool
I would say they're functionality quite equivalent, you just wouldn't pull
them in as a root volume.

You should be able to point your default StorageClass at Ceph and have it
create RBD block storage devices for you, which would auto create pvc, that
you mount you actual data in in your kube manifests.

~~~
moondev
The parent comment was asking about root volumes for kubernetes nodes

> Does Rook give you the equivalent of EBS root volumes for your nodes then?

I think we are saying the same thing if i'm not mistaken?

~~~
bogomipz
I only said "root volumes" as it's a common use case for EBS volumes. For
instance with etcd running in a container you would want the host volume to be
an EBS volume since it's critical it's a critical K8S component.

~~~
ownagefool
Right, but you would only need to mount the etcd data partition (/var/lib/etcd
afaik), rather than the entire node and/or container.

The main problem you have here, is chicken and egg. How do you use a
StorageClass, kubernetes PVC, or rook to provision Block Storage for etcd,
when you need etcd for kubernetes to function, and you need kubernetes to
function for rook, et all.

At some point, you need to bootstrap the world, which is people either start
off with cloud APIs, ansible, or PXE.

~~~
bogomipz
."The main problem you have here, is chicken and egg. How do you use a
StorageClass, kubernetes PVC, or rook to provision Block Storage for etcd,
when you need etcd for kubernetes to function, and you need kubernetes to
function for rook, et all."

I totally agree. The EBS lifecycle management is generally handled by
something like Terraform. That's why I was wondering if the use case for Rook
is primarily bare-metal Kubernetes since AWS/GCP et al. already provide these.
So I'm wondering that even in a bare-metal environment where you still need to
use config management tools like Ansible/Terraform to do things like provision
block storage what's the upside of Rook over existing iscsi/Ceph/minio
installations?

------
tyingq
Thankfully these days, places like Digital Ocean and Linode have managed K8s
where you only pay for the compute nodes, for $5/month each.

So it's fairly cheap and easy to learn on a "real" cluster without having to
build one.

~~~
gatherhunterer
Building a cluster is a fun and relatively easy project. You learn much more
by having the hardware at your fingertips. You can simulate network failures
and power failures or you can crash an important daemon. By causing problems
you can see how K8s responds and manages itself when it does not have the
nodes it expects. It is important to know these things because, for example,
if you create a pod instead of a replica set then the loss of a node will mean
the inability to use the pod assigned to that node. You need to know how a pod
and a replica set differ in order to create a self-healing stack. You can
learn all of that with the cloud solutions but the ability to answer your own
questions will always be the superior means of learning.

The Agile movement has convinced many that the solution that gets you up and
running today is the best. A good engineer is not afraid of working and
learning instead of just buying a pre-baked implementation and calling
themself an experienced user. The cluster that I know like the back of my hand
is my “real” cluster. The production cluster is someone else’s that I just
rent.

~~~
geerlingguy
Through my work on this project, I've learned a ton about kubectl, kubeadm,
the control plane, etc. that I never learned when managing EKS clusters at a
previous job. It's good to know more of the 'guts' of an insanely complex
ecosystem like Kubernetes, which is why I often recommend people follow Kelsey
Hightower's "Kubernetes the Hard Way" guide to install Kubernetes from
primitives at least once.

------
jptoto
I LOVE Jeff's work and I own his Ansible book. The setup is pretty awesome.
FWIW I think you could do this with local VMs a little more easily and
provision with Vagrant. Having said that, a cluster of Pis is super fun!

~~~
geerlingguy
What do you know, that's exactly how I do local development work for the
cluster ;) [https://github.com/geerlingguy/raspberry-pi-
dramble/tree/mas...](https://github.com/geerlingguy/raspberry-pi-
dramble/tree/master/testing/vagrant)

~~~
jptoto
Aha! Makes sense.

------
kiseleon
Just an aside, your parts list for the pi dramble has 4 Pi 4B's but micro usb
power cables

~~~
geerlingguy
Oops! I forgot to update that when I switched everything out for the 4 Bs.
I'll update that in a little bit.

~~~
geerlingguy
It has been updated.

------
Methusalah
This seems like a fun project and a great way to learn Kubernetes, but if I'm
dropping this much money on it, I'd like it to have some productive purpose
afterwards.

I'm a full-stack web dev primarily using node/react/postgres. I've also got
some projects currently hosted on a Linode instance. Ideas on fun/productive
uses for this cluster after I've built it and messed around with Kubernetes?

~~~
striking
Move the projects to your cluster and see what happens!

Consider the fact that, if you make improvements to the cluster, all of your
apps will see that same lift. So if you were to set up backups on the
cluster's persistent volumes and its databases, you'd get free backups for all
of the projects you've moved. Same with monitoring, autoscale, and so on.

------
ljm
I suppose I was lucky dabbling in one of my side-projects because I had money
to burn. I put it all in Google Cloud and then learned just how much you have
to complicate the stack (beyond the complication of K8S) to lower infra costs.

Suddenly I wasn't using basic Kubernetes any more, I was setting up a new
ingress controller so Google wouldn't launch a new load balancer instance for
_every_ public service I exposed. Those things aren't cheap.

It was an amazing way to experience just how much you can suffer in the cloud,
and just how far you can go down the rabbit hole with this kind of tech.

------
Mountain_Skies
Wonder if some old netbooks could be used for this purpose. Doubt I am the
only one with a small pile of them laying around.

~~~
ricardbejarano
From experience, anything with 2GB or more RAM can be a master node. Workers
can even have 1GB and work just fine.

Be warned, though, in my experience etcd requires a somewhat decent read/write
latency, or else it's going to fail, and when etcd fails everything fails.
Your changes don't apply, etc.

~~~
geerlingguy
_Technically_ 1GB is the limit, but yeah, at 1GB I was hitting OOM errors
every day or so (this was on the Pi 3 B+). I was very happy when I found out
the Pi 4 had 2 or 4GB models! No matter what, you should also make sure to not
schedule Pods on the master node(s) when you have limited memory (or in most
cases, really...).

~~~
ricardbejarano
Or set tight resource limits.

Setting resource requests also enables the use of HorizontalPodAutoscalers.

~~~
geerlingguy
Very true. The Pi Dramble's web pods are set to scale using the HPA.

------
acd
Thanks Jeff for your great Ansible roles!

------
madrox
This is also how my alma mater (Cal Poly SLO) teaches Hadoop. Building real
world clusters are expensive, and giving each student their own is difficult.
However, small clusters of Raspberry Pis are cheap, and it's also very easy to
demonstrate how unplugging one affects the cluster.

------
pojntfx
@alexellisuk almost all of my interest in Kubernetes is due to your work.
Thank you for everything you do!

------
segmondy
You can also build a cluster on your computer if it's beefy enough by running
multiple VMs. I bought a hp z820 workstation, 16 cores, 128gb for $1000 a few
years ago and that's my k8s experiment land.

~~~
MuffinFlavored
I don’t fully understand where the line goes from “all I need is a Docker
container for nginx + Postgres + Redis + my services” to “I need Kubernetes”

When does one need to go from just Docker containers to container
orchestration like k8s?

~~~
Nullabillity
Do you care about availability? Do you care about rolling upgrades? Do you
want to practice any of those two? Then it might be time to make the switch.

Yes, Docker has Swarm Mode. And yes, it is easier to get started with than
K8s.

The problem is that it locks you into a lot of choices when you first spin up
each (Docker) Service, that can't be changed without taking down the Service
and then recreating it with the new option. Want to change how upgrades are
rolled out? Too bad!

Kubernetes still has a few settings like that, but since (K8s) Services are
just responsible for routing traffic to separately-managed Pods, you can
always fall back to rolling the upgrade manually, or doing a blue-green
deployment.

~~~
MuffinFlavored
I do blue-green deploys with Docker and I deploy the services as “app-1”,
“app-2”, “app-3”, etc.

~~~
Nullabillity
How do you route dependencies to the correct service? If you're using a stack
then you end up needing to blue-green the whole thing, if you're not then you
still need to blue-green the whole dependent subset of the dependency graph.

And for anything exposed outside of the swarm you'd need to use some reverse
proxy, since only one service (in the whole swarm) can bind a port at any one
time.

~~~
MuffinFlavored
> How do you route dependencies to the correct service?

Reload the nginx config with proxy_pass switch from blue -> green or green ->
blue.

------
tjpd
I can also recommend MagicSandbox
([https://www.msb.com/](https://www.msb.com/)) which provides a lot of
learning content alongside a real k8s remote environment.

------
crb002
For me it was a Blue Gene/L.

------
sogubsys
Thanks Jeff!

