
$15 Production Kubernetes Cluster on DigitalOcean - discordianfish
https://5pi.de/2016/11/20/15-producation-grade-kubernetes-cluster/
======
johnwheeler
How many people are actually orchestrating a fleet of containers vs. pushing
simple apps to a single box or two with Capistrano, Maven, or Fabric? I don't
understand the popularity of Kubernetes and Docker relative to the
aforementioned simpler technologies. Certainly some organizations have use for
container orchestration, but are startups beginning with that vs. just simple
deployment scripts in hopes of scaling up? What's going on? Why are these in
the news so much?

~~~
discordianfish
Something really simple like capistrano works usually only for a small subset
of your services. With Kubernetes, you can run _everything_ in there. And you
can automate it to a very high degree: Imagine pushing a branch not only
deploys it to some test system, but it actually bring up depending databases,
a queue and a load balancing layer which then gets load tested.

This is partially possible in the cloud as well, yet it requires building
machine images which is slow. And you end up with a provider specific solution
especially if you try to integrate it very well.

But in short: It makes sense very soon if you run on bare metal (and arguably
makes it more feasible option to run on bare metal) while it makes only sense
much later if you're running in the cloud. I'd say you should have at least
more than one team deploying more than every few days.

~~~
johnwheeler
I'm not super familiar with Capistrano, but you can provision systems and
databases with Fabric. I'm still very doubtful of the utility of these more
heavyweight technologies relative to the simpler ones for all but a very small
subset of organizations. To me, it reeks of people trying to make simple
things much more complicated. I haven't used Kubernetes. I tried docker and
never understood why people would use it over an ordinary VM and some reusable
deployment scripts.

~~~
gtaylor
Kubernetes isn't just about deploying. It comes with central config store,
secrets store, service discovery, load balancing, routing/ingress, health
checks, rolling updates for your software, auto-scaling, a powerful storage
volume system, and a bunch of other things that you don't have to do a crappy
job reinventing.

------
nikon
Why does Rancher[0] never get mentioned on here? I've been using it the last
few months and it's been great. It handles everything for you without the huge
learning curve.

I'm using Gitlab.com pipelines also and it means my CI/CD pipeline is now
free, and once by build/tests are run it pings the Rancher API to upgrade the
service.

[0] [http://rancher.com/rancher/](http://rancher.com/rancher/)

~~~
chadscira
We have been using Rancher as well... It allowed us to move away from DO and
AWS. Now most of our infra is from OVH :).

It's been smooth sailing.

Because of massive costs savings we were able to just reinvest it in our own
redundancy. Also 12-factor apps are pretty damn resilient.

~~~
nikon
That's cool. I've actually just moved from ESXI hosts to OVH too.

How are you handling persistent storage? I have opted to not "dockerize"
Postgres for example after reading horror stories.

~~~
esseti
Which kind of setup do you have i OVH? do you rent servers or VMs? Do you
deploy docker? via kubernets or what? quite interested in this, if you have
material/links on OVH and docker/kube I would not mind if you can share.

~~~
nikon
As above, I have VMs on OVH public cloud[0]. Rancher can also offers an
OpenStack driver which is what OVH Public Cloud runs on, but I have not yet
tried that.

I've found certain things quite buggy with their cloud offering but the
price/performance can't be argued with.

[0]
[https://www.ovh.co.uk/cloud/instances/](https://www.ovh.co.uk/cloud/instances/)

~~~
hackerboos
OVH don't expose OpenStack APIs so don't expect that to work.

~~~
nikon
I have used the nova command line tool to manage some instances, is that not
using the OpenStack APIs?

------
vittore
So he basically followed it's the future story?
([https://circleci.com/blog/its-the-future/](https://circleci.com/blog/its-
the-future/))

~~~
tracker1
While funny, there are uses for the types of orchestration that containers can
provide... for me, it's kind of awesome that finding a script for an automated
deployment of any subject is now as simple as searching: <subject> dockerfile

And voila, I have the steps to install <subject> ... ;-) I'm being a little
snarky, but I do think that the dockerfile is pretty cool by itself, and
wouldn't even mind seeing other automation frameworks use the dockerfile and
the swarm configs for other purposes.

~~~
toomuchtodo
For sure! But please, for the love of Vint Cerf, don't go into production with
boilerplate containerization configs.

~~~
tracker1
I usually just start with the published dockerfiles and modify as needed...
Really depends on my own usage.

------
hackerboos
Flynn offers a UI installer that works with Digital Ocean.

1\. Login to DO and get your API keys

2\. Select what images you want and how many instances

3\. Click install and Flynn will setup your load-balanced environment.

4\. Push apps like Heroku

I've been evaluating dokku, Flynn and Deis. Just rancherOS left to do.

~~~
jbhatab
I would love to hear you overall review of everything. I'm really into all the
options. Dokku seems too simple for the stuff I want to do. Flynn is pretty
awesome, but I've been loving the steady progress Deis makes. Rancher is also
interesting.

I'd love to know what was the smoothest process for something like a
phoenix/node/rails app out of these. Feel free to email me at
jbhatab@gmail.com if you feel like breaking it down over chat too :D

~~~
hackerboos
I'm going to write a three-part blog posts that discusses the strengths and
weaknesses of each system. I'll email you when I'm done.

We use Cloud Foundry in production and we've used Dokku in development. CF is
similar to Dokku and Flynn as it can provide services (databases etc) and wire
them to your container.

Everyone running hobby projects should be using Dokku. It's very easy to get
started and can replace Heroku in many cases. It is missing HA but you can
always scale horizontally or load balance multiple dokku nodes (behind say
nginx/haproxy) against an external db/redis although you are essentially
running commands for each node.

Dokku should look into Docker Swarm to do orchestration but that is a mammoth
task for a pure open source project (no support business to fund it) and as
josegonzalez mentions he needs the cash to be motivated to do it. Dokku also
lacks a stable way to ship logs to external services. dokku-logspout is
available as a plugin but it frequently loses it's connection and needs to
restart. But then excels past any of the other PaaS solutions when it comes to
SSL integration via Let's Encrypt. It's a truly remarkable project.

Flynn provides services and sets up Postgres to balance between nodes so it
too is HA. Flynn works with Google, Amazon and Digital Ocean via it's web
installer (I had trouble getting it to run on a third-party provider, SSH was
broken). Flynn provides Postgres, redis, MySQL and MongoDB out of the box and
HA but with the caveat that you can't really customize these services. I also
killed one of the three nodes via the DO console and it still carried on with
no issues.

Deis was the hardest of the bunch. It essential is a manual process to get
Kubernetes up and running unless you use the cozy Google Cloud Container which
I did but isn't cheap ($100/month before bandwidth for 2 containers). Once up
and running it is as easy as the other two. They don't have any services
though - people suggested running either running Postgres in a container. I
think a PaaS should include the DB but that's debatable.

Things I haven't tried:

\- adding a node to Flynn or Deis once up and running

\- killing a node on Deis and checking if it still runs (I assume Kubernetes
takes care of this)

\- incorporating these systems with Gitlab to provide a full continuous
delivery system

~~~
josegonzalez
Dokku + Swarm actually isn't too hard (at least old swarm). I wrote a
prototype in ~10 lines of code changes this past summer.

As far as shipping logs, I feel as though thats the responsibility of the
underlying container environment, and so I'd probably have something that is
based on setting docker options for containers (or an interface to do that for
whatever scheduler you are using).

------
willcodeforfoo
This is a great guide! Very thorough. However there are so many moving pieces
it makes me think I should just use a hosted Kubernetes solution like GKE
instead of trying to roll my own...

~~~
discordianfish
Shameless plug: I also blogged about what you should roll on your own and what
not: [https://5pi.de/2015/04/22/scope-and-ownership-in-tech-
compan...](https://5pi.de/2015/04/22/scope-and-ownership-in-tech-companies/)

------
avitzurel
Continuing some of the feelings here in the comments. If you want to run
Ghost/Wordpress or any other container for that matter, there are much simpler
orchestration tools than kubernetes.

All of my blogs are hosted on a DO machine, all of them are dockerized and I
"orchestrate" them with simple chef scripts.

For this simple workload, I would go with swarm or something as simple as bash
with upstart.

~~~
vittore
That is pretty much what dokku is, with some additional bells and whistles.

~~~
avitzurel
Yeah. Point is. You don't have to go all Googly on your infrastructure, start
simple.

Deploying wordpress/ghost to DO takes 10 minutes.

~~~
josegonzalez
Actually, probably a bit less time if you use the project I recently wrote:
[https://github.com/josegonzalez/dokku-
wordpress](https://github.com/josegonzalez/dokku-wordpress)

Note: I am one of the Dokku maintainers.

------
cgag
I've been working on setting up something similar up, but it's a pain because
I'm trying out a fork of kubernetes with support for digitalocean volumes. The
author has a PR upstream and it looks like it'll probably get merged.

I've played with kubernetes on DO in the past and had a lot of trouble with
the controller node using more than 512mb of ram. Maybe I was running into a
bug, but I'd be more comfortable with a 20 dollar cluster running a $10
controller node.

I really look forward to more provider volume support landing upstream. If we
end up with support for digitalocean, vultr, and packet volumes, we're going
to be in a great place. You can run a whole lot on a 200 dollar packet
cluster. Maybe most people use more of AWS than I do, but blockstorage is
really the final thing I need kubernetes to abstract over for the provider to
not matter to me.

~~~
discordianfish
I've wrote a simple flex volume driver in bash which is included:
[https://github.com/5pi/infra/blob/master/packer/files/usr/li...](https://github.com/5pi/infra/blob/master/packer/files/usr/libexec/kubernetes/kubelet-
plugins/volume/exec/5pi.de~do-volume/do-volume) but will probably rewrite it
in golang once the new flex volume spec is final.

~~~
cgag
I actually hadn't heard of flex volumes until now, interesting.

If you're interested here's the digitalocean pr upstream:
[https://github.com/kubernetes/kubernetes/pull/36894](https://github.com/kubernetes/kubernetes/pull/36894)

------
ohstopitu
I've had a few questions about Kubernetes and production grade environments
and container orchestration.

1\. Why Kubernetes? (when mesosphere/dcos is available?)

2\. Any available best practices when setting the entire environment from
scratch? (right from server level security, to doing data backups for example)

\---

As a critic, I thought that if you were running it in production, you'd need 3
masters? (to ensure that if the master goes down, another takes over?)

So with 3 masters, and 2 slaves...the production cost go up to about $25/m. I
feel that's too expensive for what you do get (price vs performance) compared
to others (I personally feel a production environment running 5 servers from
packet.net @ ~180$/m is the way to go [0])

[0] - [https://www.packet.net/bare-
metal/servers/type-0/](https://www.packet.net/bare-metal/servers/type-0/)

~~~
old-gregg
I'll bite:

> Why Kubernetes? (when mesosphere/dcos is available?)

Frankly, if you're familiar and comfortable with DCOS, the core financial
benefit is the same: increased hardware utilization, assuming your workload
can realize that benefit. And if your application is relatively monolithic and
handles the same steady load, I'd advise to stay away from advanced
orchestration tools because they're only add moving parts and more complexity,
unless you have [1]

> Any available best practices when setting the entire environment from
> scratch?

For education, I would recommend Kelsey's excellent "Kubernetes the Hard Way":
[https://github.com/kelseyhightower/kubernetes-the-hard-
way](https://github.com/kelseyhightower/kubernetes-the-hard-way)

For production, I would recommend Telekube: [http://gravitational.com/managed-
kubernetes/](http://gravitational.com/managed-kubernetes/) It does a lot of
extra heavy lifting to make sure your k8s feels like always-on black box (I
work there).

[1] Kubernetes also has a "side" benefit of abstracting the underlying
infrastructure away, so if you have a use case of running the same complex
stack on AWS, colo, vmware, etc then running it on top of a "cloud abstraction
layer" lowers your ops/dev costs significantly. We talk about this here:
[http://gravitational.com/telekube/](http://gravitational.com/telekube/)

~~~
ohstopitu
Thanks for the links, I had to setup a Mesos cluster on ~200 machines for my
university (which is what got me interested in DCOS in the first place).

I was then trying to replicate the same in the cloud but with more constraints
(security, networking etc.) and found both Kubernetes and DCOS complicated.

Recently Azure seems to have DCOS (and Kubernetes) as it's orchestration tools
for Azure Container Service and I wanted to learn more about each.

------
cweagans
Okay, so there's 1.5GB of RAM total in the cluster. How much of that is taken
up by the OS and how much is taken up by the Kubernetes components? Can I
reasonably expect to run more than just one application on this setup?

~~~
discordianfish
Not much but this is what I'm running:
[https://github.com/5pi/services](https://github.com/5pi/services) Basically
postgres + two ghost instances. I'm actually surprised it works that well
given the constraints. With Cloudflare in front and agressive caching it even
survived this HN posting without issues (at least none I'm aware of).

But sure, it's overkill if you're not mainly interested in Kubernetes itself.

------
zokier
I notice that credential/secret management is kinda lackluster in this
otherwise pretty sweet looking setup. It would be great to have Vault thrown
in too for good measure. Maybe a followup article?

~~~
discordianfish
Yes definitely an interesting topic. Next follow up article is probably around
testing though.

------
m3adow
Very interesting. I'm currently working on a nearly identical project, except
I started what you're only considering: Using CoreOS and cloud-config (plus
some small extras). It's not done yet, but if you're interested, have a look:
[https://github.com/m3adow/coreos-kubernetes-tinc-
cluster](https://github.com/m3adow/coreos-kubernetes-tinc-cluster)

------
sandGorgon
I think k8s is "cheating" the private cloud story quite a bit. There are very
few on-metal production deployments using k8s. This is because some of the
harder problems of setting up load balancers, multiple masters, etc has not
been really solved.

in fact, just yesterday, there was a new post on the group to create a SIG-
Metal ( [https://docs.google.com/document/d/1oYtW7fgSJsQDl-
ln6ETvAQrN...](https://docs.google.com/document/d/1oYtW7fgSJsQDl-
ln6ETvAQrNdne4w_0am_qHTxtd3Yw)) because there's so less information.

load balancer integration is written for all cloud providers... but not for
metal
([https://github.com/kubernetes/kubernetes/tree/master/pkg/clo...](https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider)).
there's experimental integration with haproxy/traefik in contrib... but not in
production.

the reason is evident - a lot of the difficult parts of k8s is being
"outsourced" to the underlying cloud platform.

k8s needs to fix this fast... because the real disruption is being able to
eliminate a cloud platform (or build your own).

~~~
hubert123
that sounds bad to say the least. Isnt that the entire selling point, rolling
your own cloud... I always wondered why Azure and AWS were seemingly more of a
priority for a software solution that supposedly is made to replace the cloud
or compete with exacxtly those services.

------
pst
What people call production nowadays...

------
Axsuul
You can also use kubeadm to bring up a cluster really quickly. We use this in
production.

