
K3s: Kubernetes without the features I don't care about - ingve
https://github.com/ibuildthecloud/k3s
======
darren0
I created this (I'm the Chief Architect and Co-founder of Rancher Labs, I
experiment with stuff like this all the time). The purpose of this project is
really to embed in another project
([https://github.com/rancher/rio](https://github.com/rancher/rio) is an
example) but as a side effect it's a nice little standalone k8s package. I
never had any intention to support it as a standalone project, but hey if
somebody is interested who knows.

All of the features removed are based off of two basic ideas

1) If there are X ways to do something, I choose the best one (obviously
subjective). 2) The features are not commonly used or really shouldn't be in
k8s to begin with.

The end goal of what I'm working on is basically to use k8s code as a library
to do orchestration. Step one was to reduce the footprint of k8s (I'm 80%
there with this project), step two is to completely rework the state
management such that you no longer need to persist state in k8s. This is a
much larger goal. I've fooled around a lot with the persistence layer in k8s
(this project is running on sqlite3) so I'm basically doing a lot of work of
figuring what state I can throw away. The theory is that all desired state
comes from your yaml files. Actual state is actual (what really exists).
Everything else in k8s should be purely derived and not important.

~~~
nwmcsween
OT but why do you want to push containers? In my opinion it's a misadventure
from the real issue of a proper package manager (which in turn is a
misadventure from a proper build system) The issue with current containers is
it's the wrong level of abstraction, containment should be per binary and
ideally inline with code (see pledge(2)).

~~~
cyphar
As one of the maintainers of runc, I cannot agree with your general thrust
more (I am obviously biased since I work for a distribution company -- SUSE).
I don't agree it should be containment-per-binary, but I think that
containment should be transparent to the operating system so it can be
managed.

Currently the idea with k8s is that you just destroy anything that has broken
state and rebuild it somewhere else. While this is the easiest solution to the
problem, I think that containers should not always be treated like disposable
objects.

I think in many ways, Solaris Zones (and FreeBSD Jails to a lesser extent) did
this right. Every administrative command in Solaris is zone-aware and you can
manage your zones like you manage your host. On Linux, containers are
completely opaque root filesystems which contain who-knows-what and are
completely unmanagable or introspectable by the host. Everyone is cargo-
culting and creating hacks around this fundamentally opaque system to try to
make it less opaque (or just giving up and making it easy to purge them).

I'm working on some improvements to the OCI image-spec that _might_ help with
introspection of images[1], but runtime introspection is going to be quite
hard (needs runtime agreement).

I get that it's no longer sexy to work on container runtimes, now that
everyone is losing their mind over orchestrators (and I imagine soon they'll
be losing their mind over the next big thing). But I think that improving
these fundamental blocks is pretty interesting and might result in better
systems.

[1]:
[https://github.com/openSUSE/umoci/issues/256](https://github.com/openSUSE/umoci/issues/256)

~~~
ofrzeta
> I think that containment should be transparent to the operating system so it
> can be managed

you mean, as in "VM"? :)

~~~
cyphar
VMs are entirely opaque to the host operating system. I'm not sure what the
parallel is (though I assume you were being tongue-in-cheek).

~~~
ofrzeta
Well, I always had problems differentiating "opaque" and "transparent" because
depending on your point of view often both concepts can be applied
interchangeably.

For instance in your post I didn't really get your point because at one time
you seem to favor containers as a single entity from the perspective of the OS
so they can be easily managed. This is what I understand you called
"transparent to the OS". The is opposed to the container root fs that you call
"opaque", supposedly because it cannot be easily mounted from the host OS etc.

As the implementor of container runtimes you know that there's not a container
entity but an agglomeration of isolation techniques such as namespaces,
cgroups etc. including some kind of overlay fs that make up an actual
container. For me that's not very transparent. VMs on the other hand are in
that respect – in a similar way Solaris zones and Jails are, as they are
actual entities that can be seen and managed on the host OS.

~~~
cyphar
I mean "transparent to the OS" in reference to operating system tooling rather
than the kernel objects that make up containers (there is an argument that
these are related concepts -- but I think we can have cohesive tooling without
redesigning how Linux handles containers). The lack of a 'struct container_t'
in Linux definitely makes containers quite complicated to set up, and is a
constant source of pain. But this also allows for lots of flexibility with
different userspace programs to use various subsets of containers as part of
their operation (the Chrome sandbox is a good example of this).

> The is opposed to the container root fs that you call "opaque", supposedly
> because it cannot be easily mounted from the host OS etc.

Well, you can mount it and you can manually funge with it using 'nsenter' but
that's not really ideal. Why (from the perspective of a user) can't you run a
package upgrade across your containers? Why can't you get the free memory
across all containers? Free disk space? And so on. This information is
_available_ but it's not cohesive for _users_.

> VMs on the other hand are in that respect – in a similar way Solaris zones
> and Jails are

I agree that Solaris Zones and FreeBSD Jails are transparent (in fact that
level of tooling transparency is what I wish we had on Linux). I don't agree
that VMs are transparent in the same way -- they are an opaque DRAM blob from
the perspective of the host kernel. You can dtrace zones/jails, you cannot
dtrace a VM (and get meaningful results about what the guest kernel is doing)
AFAIK.

~~~
ofrzeta
> I mean "transparent to the OS" in reference to operating system tooling
> rather than the kernel objects that make up containers (there is an argument
> that these are related concepts -- but I think we can have cohesive tooling
> without redesigning how Linux handles containers). The lack of a 'struct
> container_t' in Linux definitely makes containers quite complicated to set
> up, and is a constant source of pain

It makes it difficult to reason about containers, as well. For instance do you
see confinement technologies such as AppArmor or SELinux belonging to the
container or are they slapped on top of containers? Where do you draw the
boundary? And does it make sense to talk about "containers" when it's such a
vague concept. Also, there are now "VM containers" such as Kata.

~~~
cyphar
> And does it make sense to talk about "containers" when it's such a vague
> concept.

Well at some point we have to be able to talk about the widely-used and glued-
together namespaces+cgroups+... concept. "Containers" is a perfectly fine
thing to discuss (especially since the history of namespaces was definitely
based in more "fully fledged" container technologies like Xen).

> Also, there are now "VM containers" such as Kata.

... which are just VMs that can interact with container tools. I'm pretty sure
that most people would not call these "containers" from an OS point of view
because they do not share any of the primary properties that containers (or
Jails/Zones) have -- the host kernel actually knows what the container
processes are doing.

------
ngrilly
The biggest issue I have with Kubernetes is it doesn't downscale. I have a lot
of projects that don't need a cluster. A single node is enough. But Kubernetes
has too much CPU and RAM overhead in such a configuration.

Because of this, I'm forced to either use another orchestrator (Docker Swarm
is lightweight enough to work on a single node or a small cluster), or pay for
machines I don't really need.

What are is the current thinking on this issue at Rancher?

~~~
falcolas
Have you considered stepping back and wondering if you really need
orchestration at this scale? Couldn't a simple unit file suffice?

If you have many such projects, perhaps something like Saltstack or Ansible,
both of which have support for Docker container management built in.

~~~
ngrilly
Great question!

I already use Ansible to setup machines and automate deployments.

I used systemd unit files until I switched to Docker. I don't use them anymore
since dockerd supervises the container itself.

I use Docker Compose to describe the set of services composing my app and it's
great.

But in production, Docker Compose is unable to provide zero-downtime
deployments, and this is why I'm looking at Docker Swarm or Kubernetes.

systemd unit files are not very useful for Docker containers, and they don't
solve the zero-downtime deployment issue.

------
xrd
This is such a great stepping stone to playing with k8s. Maybe there should be
a book with each chapter named k1s, k2s, k3s. Each chapter introduces more and
more features with the why of each. Eventually you get to the full set of
features and can use k8s in its entirety.

~~~
qohen
Speaking of "a great stepping stone to playing with k8s", there's a free,
scenario-based site called KataCoda.com out there that gets you putting
Kubernetes (and Docker and other things like TensorFlow) through its paces --
no need to install anything, it's all browser-based.

The site also has some "playgrounds" for experimentation once you know what
you're doing and apparently can be used to create one's own courses, e.g. for
training people at your company, etc.

List of all the courses:
[https://www.katacoda.com/learn](https://www.katacoda.com/learn)

------
vemv
Presumably some of those features will be relevant to you as you scale.

Also, ignoring a feature seems far cheaper than forking a project and
renouncing to updates for desired features?

e.g: feat A is desired, feat B is not, a new upstream refactor touches A and
B. How do you stay up-to-date?

------
stevenacreman
Curious if this would be better than minikube when running in a CI pipeline to
run tests on.

------
mosselman
For years now my favourite way to deploy my own 'side-projects' has been
Dokku. It is like having your own Heroku on a single machine. Very neat, but
no clustering.

Lately I have been playing with kubernetes and now docker swarm. I must say
that I too find kubernetes to be too heavy (feature- and concept-wise) for the
smaller scale things that I am up to: I have to read up too much, learn too
many new concepts, write too many config files.

Docker swarm is a lot more appropriate for scaling those side-projects to a
slightly bigger step, however, I have concerns about the longevity of the
project. Will Docker Swarm still exist in 5 years?

------
pcmaffey
I can't wait for someone to come out with a product called `k8OS`

------
eganist
What do you have against Rnete?

~~~
darren0
Against Kubernetes? Nothing, it's a beautiful architecture. I just want to use
the code and architecture in different ways and specifically with lower
resource constraints.

~~~
eganist
Oh, I was merely making light of the name "kubes" which I figure is what you
meant with the numeronym. Apologies that my dog couldn't hunt.

~~~
chrisseaton
> Apologies that my dog couldn't hunt.

Huh?

~~~
troygoode
Its an American English idiom, in this case referencing his joke not
performing the intended job of making you laugh.

[https://english.stackexchange.com/questions/52755/meaning-
an...](https://english.stackexchange.com/questions/52755/meaning-and-origin-
of-that-dog-dont-hunt)

~~~
wink
this is getting real meta now :)

Thanks for the explanation.

------
bboreham
I salute the enormous amount of work you’ve done. But without a consistent
store, it seems to me there are a myriad of race conditions the system can
fall into.

Maybe if you accept a bit of oscillation under stress, and damp that
oscillation a lot, it would be ok.

------
infogulch
Not sure what it is, but the name really tickles me. Bonus points that it's
kinda meta.

~~~
geoah
It's a cut-down version of kubernetes. Author left the basic features and
removed most things that are oriented to heavier users or that required more
resources than they wanted to spare.

I assume the goal is to make k3s into something light where you can deploy
small projects instead of using something like swarm/dokku.

He has also removed etcd and replaced it with sqlite which seems like a nice
way to reduce resource requirements.

------
crb
I more than get the joke, but I'm worried about the ambiguity of "5 less then
k8s. Short for kates, pronounced k8s", and worry that it will stop you from
getting successful, if that's your intention.

Are you saying your project is named "kates"? Why not name it something like
"Khios", a Greek island, which still lets you keep 'k3s' as your abbreviation?

~~~
darren0
I never intented anyone to directly use it. The "selling point" is what I'm
going to put on top of this. This is intended to be an internal library. Hence
the stupid puny name.

~~~
dsnuh
How about "k2s", short for "K.I.S.S."? So simple it saves you an S. :)

