
Kubernetes – The Hard Way - yarapavan
https://github.com/kelseyhightower/kubernetes-the-hard-way
======
rmoriz
I'm surprised that RedHat's OpenShift 3 / Origin does not get any publicity on
HN. It uses Kubernetes (of which RedHat silently became the largest committer)
and also fills several missing parts including a CD pipeline.

[https://www.openshift.org/](https://www.openshift.org/)

It's amazing to see how many large Enterprises already selected OpenShift,
either in various cloud setups or on premise scenarios.

Looks like that for many HN people there is only AWS and GCE out there.

~~~
ogrisel
Apparently the goal of this guide it to setup a kubernetes from scratch (on
top of VMs provisioned on an orchestration-agnostic public cloud), not how to
use a managed kubernetes cluster as provided by openshift.

~~~
avtar
Perhaps the parent was referring to the self-hosting OpenShift option which
acts as a Kubernetes distribution:

[https://docs.openshift.org/latest/getting_started/administra...](https://docs.openshift.org/latest/getting_started/administrators.html#getting-
started-administrators)

------
cies
Cannot find much on mounting storage to the container. It exactly that
--storage-- that has been recognized as a difficult problem when moving an
infrastructure to containers.

Given the title "The Hard Way", I kind of expected it would dive deeper into
this topic.

~~~
jordanthoms
Of course - following the first law of technical documentation, the amount
written about it is inversely proportional to the complexity of the problem.

Seriously though, it is getting a bit silly how almost every single kubernetes
demo involving persistent data just uses emptyDir or hostPath and we are all
supposed to pretend that those are a production-ready solution that won't lose
data. AFAIK the PersistentVolumeClaim and StorageClass stuff will be the
solution for this?

~~~
cies
To me it seems that there are many approaches. A storage cluster seems to be a
necessity for anything serious. But then the questions is: which one? Open
source solutions are not too many (Gluster, Ceph, FreeNAS come to mind), and
from the closed source camp ScaleIO looks really promising.

But non of these are part of your K8s tutorial :)

~~~
creshal
And gluster at least has horrible performance and stability issues. And with
containers, kernel panics kill an entire host…

~~~
strikerz
Do you have more details on performance and stability issues? The recent
releases of gluster have been working very well.

Gluster is completely in userspace. How would stability issues, if any, in
userspace cause kernel panics? Can you comment more about that?

~~~
creshal
> Do you have more details on performance and stability issues?

Tried using it for a while (last November until ~April) on Debian Jessie, with
latest versions. Lots of bugs with SSL mode, lots of desynchronization issues,
crippling performance issues, …

And no matter what I ran into, the bugs were already known and filed for
months at that point, with zero developer reaction.

> Gluster is completely in userspace.

Unless you try using its NFS server to get not _entirely_ disastrous
performance.

~~~
strikerz
What workloads were you using Gluster for? How did you reach out to the
developers? Usually the mailing lists are very responsive and most bugs
brought up on the lists are addressed.

> Unless you try using its NFS server to get not entirely disastrous
> performance.

Wrong again, Gluster's NFS server is in userspace too. Not sure how that can
cause a kernel panic.

~~~
creshal
> What workloads were you using Gluster for?

Sharing config files. Wordpress sites. Django sites. The performance was too
shitty for everything.

> How did you reach out to the developers? Usually the mailing lists are very
> responsive and most bugs brought up on the lists are addressed.

IRC. I ended up finding all my errors unresolved by it in the mailing list
archives with zero developer attention, and since I didn't have the time to
help RedHat make their product actually usable, I dropped Gluster.

> Not sure how that can cause a kernel panic.

I guess it's technically not a kernel _panic_ if I/O just stalls so hard that
you can neither mount nor unmount nor otherwise access _any_ of your NFS
targets… but you have to hard reset the whole machine anyway.

------
okket
Previous discussion:
[https://news.ycombinator.com/item?id=12323187](https://news.ycombinator.com/item?id=12323187)
(23 days ago, 79 comments)

~~~
yarapavan
Thanks!

Please see author's tweet here -
[https://twitter.com/kelseyhightower/status/77498377095689830...](https://twitter.com/kelseyhightower/status/774983770956898307)

The github repo is updated with updated DNS add-on, better examples and AWS
support.

~~~
justinsb
Please note, this is not a supported AWS configuration. This is just a way to
run Kubernetes on AWS instances, but it is really just treating them as bare-
metal.

My advice: You should be using kops or kube-aws for production. (I work on
kops, so I am biased to believe it is important)

~~~
yarapavan
Thank you! Learned about kops today.

------
brobinson
Is there a "Kubernetes - The non-Google Way"? It seems like every tutorial is
for GCE.

~~~
kelseyhightower
I've updated the tutorial to include AWS a few days. Based on your feedback
I've added a note regarding AWS support toward the top of the project README
and in the project description field.

In some cases you'll find side-by-side labs that focus solely on AWS like this
one for bootstrapping the underlying compute:
[https://github.com/kelseyhightower/kubernetes-the-hard-
way/b...](https://github.com/kelseyhightower/kubernetes-the-hard-
way/blob/master/docs/01-infrastructure-aws.md)

In other parts of the tutorial you'll find sections for both GCE and AWS that
highlight the different commands required for each platform - only a small
fraction of the tutorial requires something different between GCE and AWS.
Both cloud providers are now treated equally in the updated tutorial.

~~~
brobinson
Thanks, Kelsey!

------
wyclif
The very first Lab link gives a 404 error. Heads up.

~~~
gdubya
That's why it's "the hard way"

------
justinsb
Please note that this guide is _only_ for learning purposes - it makes a lot
of decisions that do not follow the recommended production approach. If you're
installing for production purposes, you should use one of the existing tools
if you can (I work on kops and so naturally recommend it for AWS). If none of
the tools meet your needs, you should probably first open issues on those
tools; if you still must write your own installation method, you should
combine this guide with the official documentation to understand a cluster as
created by one of the recommended tools, before building your own.

~~~
kelseyhightower
Happy to improve the tutorial to incorporate the production approach that
should be taken for each task. The main goal is allow people to run through a
cluster bootstrap step by step so they learn how things work.

If you don't mind filing a few issues on the project I'll be happy to rework
each lab to start addressing these concerns. Until then I've added a note to
the README regarding production readiness, but that should not prevent people
from learning.

~~~
justinsb
You should participate in sig-cluster-lifecycle. The conclusion of that sig
(or my interpretation of it) was that having a production suitable cluster is
- today - essentially orthogonal to having a discoverable build-your own
cluster. So we are all best served by having this document for those that want
to learn about how kubernetes works, but a separate approach for installing a
production cluster that prioritizes correctness over obviousness. I think what
you are doing is great, but I worry that using words like "support" imply that
you will be offering some support for people that try to run this in
production.

However, the long term goal of sig-cluster-lifecycle is to bridge the gap
here, to make a production configuration easy and obvious. So I'd love to see
you start using the new kubeadm work, for example, so that we can meet in the
middle!

~~~
kelseyhightower
Thanks for the feedback. My goal is to augment those tools with step-by-step
documentation so people get an opportunity to learn how things work. Yes, I
will attempt to "support" people as they attempt to learn things the hard way.

I would also like to note that I'm not in competition with those automation
tools. I'm just really focused on helping people learn. Some parts of the
Kubernetes community really want to learn this stuff without an automation
tool so they can skill up to build their own, custom, tools that match their
preferences and tradeoffs.

~~~
justinsb
That's wonderful - I think what you're doing is super important - everyone
wants configuration to be easy enough that the automation tools are a
comparable effort to doing things yourself! Helping people that go the
kubernetes-from-scratch route has proven really difficult in the past, hence
the focus on making things easier before encouraging manual installation, but
I'll start sending people your way :-)

