
Why Kubernetes is winning the container war - bdburns
http://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html
======
avitzurel
I've used both mesosphere and Kube now (in production) and I feel I can safely
comment on this.

Kube is winning for the same reason React/Redux (and now Mobx) is winning and
why Rails was winning at the time. Community.

The community for Kube is awesome and the work people are doing in the field
is being noticed all over the place.

I've seen people (myself included) that moved production clusters from mesos
to Kube just because the activity of the development and how secure they feel
with the community and the project going forward.

React and Rails (at the time) had the same sort of pull towards the community
and why a lot of people on-boarded.

Golang is most likely a factor here too. I feel most people find Golang
friendlier than Scala/Java. That's why Kube has many more contributors, the
hurdle for contributing is easier to jump

~~~
runT1ME
>I've seen people (myself included) that moved production clusters from mesos
to Kube just because the activity of the development and how secure they feel
with the community and the project going forward.

This is a bit disappointing and disconcerting to hear! I understand that
technology doesn't happen in a vacuum, but when you have two well
supported/known technologies, picking the most 'popular' one rather than
something based on technical merit is a huge issue for me.

~~~
hosh
The popularity is great, but Kubernetes also happens to have an awesome
technology too. Where it intersects with the community is that the technical
direction actually listens to real-world use-cases that members of the
community brings up. Combined with the amount of activity, Kubernetes is
evolving fast.

For example, the PetSet feature of Kubernetes 1.3 involved a really long
discussion with people trying to work out how and why to use it.

I remember back in the Kubernetes 1.1 days when I was trying to get AWS ELBs
to talk with Kubernetes and possibly finding drivers to automatically set up
proxy points. In my search, I found the Github issues where people were
discussing that very thing. The person who wanted AWS integration was there
talking about the specific needs and quirks of the AWS ecosystem. There were a
series of patches made to beef it up, and the discussion on Github documented
how the feature was conceptualized (so as to be part of a more general
solution on port management).

In contrast, I don't see that with Docker. I can't say anything about Mesos,
as I've never tried using it.

~~~
avitzurel
I agree, that's exactly the proof of a strong community.

Community goes a long way but has to co-exist with strong technology and being
open to change.

------
tzaman
For a devOps fan like me, k8s has been a godsend, and what I like in
particular is their 3 month release schedule. There are still some hiccups
like no good documentation (or a tutorial really) on setting up shared
writeable storage and how to handle databases, or more importantly
replication.

The k8s team is very responsive and I'm sure these will be ironed out in the
near future so we can all adore each other's cattle :)

~~~
yeahbutbut
How did you handle shared writable storage and databases?

~~~
ex3ndr
We (actor.im) also moved from google cloud to our servers + k8s. Shared
persistent storage is a huge pain. We eventually stopped to try to do this,
will try again when PetSets will be in Beta and will be able to update it's
images.

We tried:

* gluterfs - cluster can be setup in seconds, really. Just launch daemon sets and manually (but you can automate this) create a cluster, but we hit to that fact that CoreOS can't mount glusterfs shares at all. We tried to mount NFS and then hit next problem.

* NFS from k8s are not working at all, mostly this is because kubelet (k8s agent) need to be run directly on a machine and not via rkt/docker. Instead of updating all our nodes we mounted NFS share directly to our nodes.

* PostgreSQL we haven't tried yet, but if occasional pod kill will take place and then resyncing database can became huge issue. We ended up in running pods that is dedicated to specific node and doing manual master-slave configuration. We are not tried other solutions yet, but they also questionable in k8s cluster.

* RabbitMQ - biggest nightmare of all of them. It needs to have good DNS names for each node and here we have huge problems on k8s side: we don't have static host names at all. Documentation said that it can, but it doesn't. You can open kube-dns code it doesn't have any code at all. For pods we have only domain name that ip-like: "10-0-0-10". We ended up with not clustering rabbitmq at all. This is not very important dataset for us and can be easily lost.

* Consul - while working around problems with RabbitMQ in k8s and fighting DNS we found that Consul DNS api works much better than built-in kube-dns. So we installed it and our cluster just goes down when we kill some Consul pods as they changed it's host names and ip. And there are no straightforward way to fix IP or hostnames (they are not working at all, only ip-like that can easily changed on pod deletion).

So best way is to have some fast(!) external storage and mount it via network
to your pods, this is much much slower than direct access to Node's SSD but it
give you flexibility.

~~~
lobster_johnson
As long as you associate a separate service with each RabbitMQ pod, you can
make it work without petsets. (Setting the hostname inside the pod is trivial,
just make sure it matches.) Then you can create a "headless" service for
clients to connect to, which matches against all the pods.

If you set it up in HA mode, then in theory you don't need persistent volumes,
although RabbitMQ is of course flaky for other reasons unrelated to Kubernetes
-- I wouldn't run it if I didn't have existing apps that relies on it.

~~~
ngrilly
> RabbitMQ is of course flaky for other reasons unrelated to Kubernetes -- I
> wouldn't run it if I didn't have existing apps that relies on it.

I'm surprised because I know teams which are very satisfied with running
RabbitMQ at scale. Could you elaborate?

~~~
lobster_johnson
RabbitMQ doesn't have a good clustering story. The clustering was added after
the fact, and it shows. I've written about it on HN several times before, e.g.
[1]. Also see Aphyr's Jepsen test of RabbitMQ [2], which demonstrates the
problem a bit more rigorously.

With HA mode enabled, it will behave decently _during_ a network partition
(which can be caused by non-network-related things: high CPU, for example),
but there is no way to safely recover without losing messages. (Note: The
frame size issue I mention in that comment has been fixed in one of the latest
versions.)

We have also encountered multiple bugs where RabbitMQ will get into a bad
state that requires manual recovery. For example, it will suddenly lose all
the queue bindings. Or queues will go missing. In several cases the RabbitMQ
authors have given me a code snippet to run with the Erlang RELP to fix some
internal state table; however, even if you know Erlang, you have to know the
deep internals of RabbitMQ in order to think up such a code snippet. There
have been a couple of completely unrecoverable incidents where I've simply
ended up taking down RabbitMQ, deleted its Mnesia database, and started up a
new cluster again. Fortunately, we use RabbitMQ in a way that allows us to do
that.

The bugs have been getting fewer over the years, but they're not altogether
gone. It's a shame, since RabbitMQ _should_ have a model showcase for Erlang's
tremendous support for distribution and fault-tolerance. You're lucky if
you've not had any issues with it; personally, I would move away from RabbitMQ
in a heartbeat, if we had the resources to rewrite a whole bunch of apps.
We've started using NATS for some things where persistence isn't needed, and
might look at Kafka for some other applications.

[1]
[https://news.ycombinator.com/item?id=9448258](https://news.ycombinator.com/item?id=9448258)

[2] [https://aphyr.com/posts/315-jepsen-
rabbitmq](https://aphyr.com/posts/315-jepsen-rabbitmq)

~~~
ngrilly
Thanks a lot for elaborating. This is exactly the kind of insights I wanted to
know.

------
amr_abdelrazik
Disclaimer: I work for Mesosphere (Champions of Apache Mesos and DC/OS)

We have total respect for K8s, but I don't think you can claim winning just
based on the community and stars.

OpenStack has a huge larger community of developers and advocates, but it is
still haven't reached it's potential despite many years and incredible effort,
seminars and summits

Also most of these next gen infr projects now (DC/OS which is powered by
Mesos, K8s, docker swarm ) are converging feature wise, but they also have
their strengths and weaknesses, some of these are temporary, some of these are
structurally by design

Mesos (and DC/OS) were not just designed for scale, but also for extensibility
and different workloads, which is why you can run Cassandra, Kafka, Spark,
..etc In production. None of these workloads run as traditional containers,
with DC/OS they have their own operational logic such as honoring placement,
simple installation, and upgrade, which is a core design function of the two-
level scheduler

People usually complain that standing up Mesos is hard, which is why we we
built DC/OS and open sourced it to give you the power of mesos and its awesome
capabilities in managing containers and data workloads without going through
all the effort to stitch everything yourself. Check it out at DCOS.io, I am
sure you guys will be blown away.

~~~
eikenberry
The problem I always here with Mesos is that it doesn't scale down well. I.E.,
that it is not developer friendly or, to put it another way, you can't run it
on your laptop.

~~~
SEJeff
Actually that isn't true. Minimesos runs quite well on your laptop if you wish
to test that way:

[https://minimesos.org/](https://minimesos.org/)

------
leetrout
I know it's young still, but I think Nomad is going to get a share of this
market with little effort.

I played with Mesos & k8s and I picked Nomad instead. Now I'm not managing a
huge fleet of servers I want to abstract away as much as I wanted a simple,
consistent framework for allocating resources to containerized tasks across a
small group of nodes. And I don't think that use case is anything to sneeze at
and for a new user there just isn't anything out there as easy as nomad IMO.

[https://www.nomadproject.io/](https://www.nomadproject.io/)

~~~
smnscu
For a simple solution, also see Rancher. I've used them about 2 years ago and
even then it was a very useful and stable tool.
[http://rancher.com/](http://rancher.com/)

~~~
loopbit
I also use Rancher and quite happy with it.

The cool thing is that you can set up and environment using cattle (their own
orchestration system), k8s, mesos or swarm. Up to you and your preferences.

------
batmansmk
It pretends to compare Kubernetes, Apache Mesos and Docker Swarm. This article
says Kubernetes has a lot of stars on github (doesn't compare it to Docker or
Mesos, only says Kubernetes has a lot), same for Slack/Stack Overflow and
number of CVs mentioning the tech ... I will pass Infoworld opinion from now
on.

~~~
aCandidMind
I completely agree that this is an article only intended to praise Kubernetes.
Stating Google invented Linux containers alone is wrong as well.

------
virtualnm
I can bring up an app on Linux or Windows from bare metal in minutes by hand.
But the way it's supposed to be done now is something like this, right:

    
    
      1) Use one of Chef/Puppet/Salt/Ansible to orchestrate
      2) One if those in item 1 will use Vagrant which
      3) Uses Docker or Kubernetes to
      4) Set up a container which will
      5) finally run the app
    

Really?

~~~
swozey

      1) Developer pushes code to repo, its tested, if pass>
      1) Rkt/Docker image made by CI/CD system and pushed to docker registry
      2) Automatically deploy new image to staging for tests and master for production (can be a manual step) 
      3) Sit back and relax because my time invested up front saves me hassle in the future
    

I have 30+ nodejs apps that can be updated on a whim in under 5 seconds each.

Setting up bare metal instances, even when using something like ansible is
slow and cumbersome compared to what I'm now doing with k8s

Doing an apply/patch to kubernetes nodes takes seconds (assuming services
aren't affected).

edit: Sorry, for the unfamiliar by services I mean load balancers/ingress,
it's a k8s term. It takes 45s-1minute to update glbcs so I modify them as
rarely as possible.

~~~
daxorid
How do you solve your #2?

We push our docker images to quay.io but pull+deploy is still manual. How does
the target platform detect a new container on the registry in order to kick
off a pull+deploy?

~~~
lobster_johnson
Take a look at Drone ([http://readme.drone.io](http://readme.drone.io), not to
be confused with the hosted Drone app). It allows you to execute arbitrary
commands after publishing the Docker image. You can tell it to run kubectl to
perform a deploy, for example.

Drone can also run CI tests and build stuff in temporary containers, which
avoids the need to compilers etc taking up space in the final image, and
negates the need for squashing (which busts your cache).

Much faster than Quay. Quay starts a new VM in AWS every time it builds, which
is super slow. Drone just starts containers.

(You have to self-host Drone, though. Since it requires running Docker in
privileged mode it's not a good fit for running under Kubernetes,
unfortunately.)

(Kubernetes on GKE always runs in privileged mode, but that doesn't mean it's
a good idea!)

~~~
robszumski
Quay.io has been beta testing a new build system (based on Kubernetes under
the hood) that should make builds a bit faster. If you're interested in
testing it out, your organization can be whitelisted. Tweet at @quayio and we
can get you set up.

------
stcredzero
_It 's all about knowing how to build an open source community_

This. Engineering excellence is secondary. You can get away with complete
craptitude in your tech if you can build community. (I won't name examples.)
Of course, it's better if you also have technical excellence. On the other
hand, you can have technical excellence, but it will come to naught if you
have community destroying anti-patterns.

------
mr_luc
In my experience, I haven't been coming to k8s because I particularly like the
developer experience (despite their efforts to focus heavily on it), but
because it cleanly supported some things that I need.

For instance, with k8s, out of the box every running container in a clustered
system is discoverable and has its own IP. If you're writing distributed
applications, and you're using containers principally as a tool to make your
life easier (and not as part of an internal paas or for handling data
pipelines or some other use case), having that sort of discovery available out
of the box is great.

------
_asummers
One thing I've found extremely difficult to handle is the Zookeeper cluster
model of containers. Where when a thing dies, a thing has to come back and be
able to referred to as "zookeeper-1" forever. The way to do this currently is
to use a service in front of a replication controller with one pod. This feels
wrong all over. Supposedly they have a thing called Pet Sets [1] coming to
solve this, but it's been in the works for an eternity. Also we've started to
outgrow the load balancing simplicity that the k8 load balancer gives you, and
I have not seen a nice migration path to something like HAProxy in Kubernetes.
All that said, we like kubernetes a lot.

[1] To distinguish from cattle. If you have a red blue and green goldfish, and
your red goldfish dies, you can replace with another red fish and not really
notice, but if it's purple, the others won't play with it.

~~~
jat850
Re: load balancing. Out of curiosity, why not just use haproxy? Or nginx, or
your load balancer of choice.

Run haproxy in host networking mode, as a daemonset, and use it as your
ingress point?

~~~
_asummers
So it would basically be daemon set sitting on top of all the other services,
with their own load balancing turned off? That should work. I like that

~~~
jat850
Email's in profile. I'm happy to discuss specifics offline, as I have some
experience with this. Can show some example configs of how we've employed it.

------
avitzurel
One thing that I think Kube (and dc/os) are missing is what Chef is working on
right now. Application definition should live within the app and consumed by
the scheduler.

Chef's product is called Habitat
[https://www.habitat.sh/](https://www.habitat.sh/) and it has some VERY
interesting concepts that if Kube will implement it will be much more
interesting (to a lot of people).

Right now, the deployment and configuration of an application is supposed to
be separated but I feel they need to be just a bit more coupled. An engineer
that develops the application will define a few things like "domain" and how
you connect to the application and this will be consumed by the scheduler.

Right now, dc/os and mesos are really fine grained around the DevOps people
and I feel that the first that will crack the "batteries included" approach
will win the fight by a knock-out.

Imagine something like Heroku on your own infrastructure, if an engineer can
launch and deploy a micro-service with the same ease they deploy and access a
Heroku application. That will be awesome.

~~~
lobster_johnson
While coupling is convenient, there is a problem when you are not the only
consumer of your code. If you have an open source project that is to be
deployable by anyone, you can't put Kubernetes files there (there's not really
a way to write a generic Kubernetes manifest -- in addition to volumes and
secrets, things like labels are environment-specific). This is also true for
other things like .drone.yml for Drone.

So you have to maintain a private "release fork" that you periodically rebase
into. This becomes a workflow bottleneck, and the VCS history will not look
good, either. Even worse if you have multiple branches you deploy from.

~~~
avitzurel
This is true. It's especially true if you have services depending on one
another. I think there's a middle-ground there.

Right now, 99% of people I know use Github-Workflow, master is always
deployable (often CD), other branches are able to deploy using "hubot deploy
branch-name to xxx"

I think if you take the latter and you make it easier to deploy an application
on top of your cluster, you can gain a lot.

The workflow today is more cumbersome. If you could create your application,
attach some definitions to it and launch it to the scheduler, it would be
better.

------
sebringj
Just as an outside observer developing on a platform, I see my fellow devOps
team members working in Kubernetes and it's been a shit storm for the most
part where containers disappear randomly, stuff breaks and they are on call on
the weekends not having fun. I have my own clients on other projects using AWS
where I just upload and click buttons like a dumb monkey and I end up looking
more competent even though I'm completely not. I've consequently not been
motivated to dive into these DIY deployments just yet.

------
spudfkc
I haven't dived into Kubernetes yet, but I set up Rancher for our new
application and it has been nothing short of amazing so far. I can't express
how happy we've been with it.

I previously tried the Mesos/Marathon route (with Mesosphere and then again
with Mantl) and that was nothing but a huge waste of time due to all the
maintenance that was necessary for all the required servers. With Rancher,
it's just spin up a container for each host with a single command and you're
done.

~~~
webo
I've been looking at Rancher for a few weeks, haven't put an application yet.
How do you perform rolling updates with Rancher+Kubernetes? With Kubernetes
itself, currently I just do `kubectl rolling-update <rc> \--image <new-image>`

------
pawadu
From the subtitle:

 _It 's all about knowing how to build an open source community -- plus
experience running applications in Linux containers, which Google invented_

~~~
nullspace
It kinda did, right? Maybe, "invented" is to bold of a term, maybe there were
others before. But, I've heard folk tales that Google has been running stuff
in containers since the early to mid 2000s.

~~~
lern_too_spel
Google contributed cgroups to Linux, so saying that it invented Linux
containers is a fair statement. Other operating systems had containers before
Linux.

~~~
SEJeff
Not exactly. Google contributed the namespace support to Linux and IBM
contributed much of the initial control group bits to Linux. Together, they
form the modern "container" that we have all came to know and love.

------
0xCMP
The setup to get k8s running isn't great, but once it's running and you
understand it's config files it makes things so much easier. We're getting
ready to deploy k8s at work soon and begin moving more there as we can.

From what I understand, and is completely not in the article, Mesos is
designed for scale while most start-ups (and even established companies) can't
afford or justify. K8s is simpler but still robust. Better than just fleet or
compose and clearly still better than swarm (based on posts read here on hn).

~~~
steilpass
Disclaimer: I work for Red Hat.

> The setup to get k8s running isn't great,

+1 But they are greatly improving.

For a local test environment have a look at OpenShifts 'oc cluster up'
[https://github.com/openshift/origin/blob/master/docs/cluster...](https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md)

~~~
brazzledazzle
Someone elsewhere in this thread complained about the NFS experience with k8s.
I know OpenShift contributed some or all of that code so does it improve the
NFS experience as a layer on top of k8s?

~~~
joefern1
Disclaimer: I work at Red Hat on OpenShift.

All of the code Red Hat contributed around Kubernetes storage plugins went
into Kubernetes upstream. If you have any questions or problems, feel free to
raise an issue or reach out to us in the community. Red Hat has a large number
of contributors on the Kubernetes project and we are big fans of the community
and the technology!

Red Hat OpenShift provides a an enterprise-ready Kubernetes distribution that
also builds on top of Docker, RHEL/Fedora/CentOS and includes integrated
networking (OVS), routing (HAProxy), logging (ELK stack), metrics, image
registry, automated image builds & deployments, integrated CI/CD (Jenkins),
and self-service UX (Web, CLI, IDE). You can check out the free Origin
community distro here -
[https://github.com/openshift/origin](https://github.com/openshift/origin) or
sign up for our free hosted developer preview here:
[https://www.openshift.com/devpreview/](https://www.openshift.com/devpreview/).
We also offer commercially supported solutions -
[https://www.openshift.com/container-
platform/](https://www.openshift.com/container-platform/).

------
DubiousPusher
Seems like "Why Kubernetes is winning the orchestration war" is a more
appropriate title for this?

------
technologia
I use Mesos and K8s heavily as well as contribute back to the projects, and
while I do agree this is leaning towards being fan-fare there is a bit of
truth to this.

Community is a big deal, people tend to underestimate this; Putting aside
newer companies, when a larger enterprise ventures out to open source they do
take community as a major factor as you have to consider the tool you build
will have to last 5 years, maybe 10, maybe more.

To delve further into the Docker side of things, I personally wish that the
company would focus on its core business instead of stretching itself with
extra things. I get the need to improve the UX, which they do very well
considering how far we have come from LXC.

I feel Mesosphere starting to go down the feature creep route as well, but I
wish them all the best as I loved Mesos since the beginning all those years
ago.

------
aCandidMind
Why is no one mentioning Docker's response with version 1.12, the new built-in
orchestration called swarm mode (different than just swarm):
[http://blog.nigelpoulton.com/docker-launches-kubernetes-
kill...](http://blog.nigelpoulton.com/docker-launches-kubernetes-killer/)

Granted, the title is fanboyish, but it really seems to be a significant
response to Kubernetes.

------
contingencies
IMHO nobody is winning anything that matters right now because the current
transition is a transition to an additional level of abstraction which is
_definitely_ not properly met by any of the tools available.

What we now need is _tools that allow architectural fences around components,
and reliability guarantees around subsystems_ ... versus not only technical
but also business-level risks (including state-level actors)... often across
borders, including for example exchange rate risk. This is based on business-
level risk models, not some engineer feels X or Y type reasoning which is
(often very well, but broad-picture uselessly) based on technical know-how.

I prototyped such a system pretty successfully, you can read the design @
[http://stani.sh/walter/cims/](http://stani.sh/walter/cims/) .. it's
incomplete (critically hard to explain investment utility for non-tech
business types) but at least infrastructure agnostic and practically informed.

NB. To be non-humble, by way of demonstration I am a guy who has been called
to conceive/design/set up/manage from-scratch datacenters for extremely
visible businesses with highly motivated attackers with USD$Ms at stake for
penetration. Systems I personally designed run millions of USD/month and have
many times that in investment. And it's obvious that both Google ("compete
with amazon on cloud business... high-level desperate for non-advertising
income!") and Docker ("VC! growth! growth! growth!") have their own agendas
here... we should trust no-one. It's early days. Bring on the ideas, build the
future. Hack, hack, hack. We the little people own the future. It's ideas.

------
huntc
Looking through all of these comments we should be glad, as a community, that
there are a number players in this burgeoning orchestration space. Nobody has
won and nobody should win. They each have their strengths and weaknesses and
there is no one size that fits all.

------
graffitici
Sad to see Mesos losing steam. My understanding was that Mesos subsumes the
functionality of Kubernetes thanks to its Aurora scheduler. But it has much
more customized schedulers, for different purposes, that might make it more
efficient to run complicated pieces of software.

For instance, it certainly is possible to run a Cassandra cluster by having
each instance run in its own Docker container. My understanding is that it
would be much more efficient to run this cluster with a dedicated Cassandra
cluster instead.

Is this right? Or are the performance benefits of running a dedicated
Cassandra scheduler on Mesos negligible compared to running them in
containers?

------
applepple
It would be nice if open source projects (especially popular databases) came
with k8s definition files so that you wouldn't have to write the yaml
yourself.

~~~
jondubois
I did that for my open source project SocketCluster. See
[https://github.com/SocketCluster/socketcluster/tree/master/k...](https://github.com/SocketCluster/socketcluster/tree/master/kubernetes)

Also, if you look at the main Kubernetes repo on GitHub, you can find the yaml
files for various popular open source projects in there as well. Some of them
are out of date though - Things have been changing so fast.

I think a problem that still comes up is that you still need yaml files to
account for the interactions between different services too. Each service
doesn't live in a vacuum.

We're currently working on building an complete auto-scalable stack on top of
Kubernetes and a deployment service to go with it. See
[https://baasil.io](https://baasil.io)

Everything we do is open source except the actual website.

~~~
hackitself
I use SC for my startup and it works great! Thanks. I'll have a look at the
Baasil service, it sounds interesting.

------
paimpozhil
I wish kubernetes has more examples for example their vsphere volume driver
has almost 0 documentation/tutorials on how to set that up.

[http://kubernetes.io/docs/user-guide/volumes/#vsphere-
vmdk-e...](http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-example-
configuration)

I believe is inadequate

~~~
smarterclayton
Agree - better doc is a focus for us. We've added providers so fast the doc
has not caught up.

------
shadykiller
The author is just gaga over google. He presents no comparison, no benchmarks
and no mention of alternatives except docker swarm.

------
iagooar
What is the best resource to learn Kubernetes like a boss? I like ebooks, but
will take anything, as long as it's up-to-date and easy to follow without
being a long-term sysadmin.

~~~
jat850
You might not like this answer, but... play with it. In our environment, we're
compiling and deploying Kubernetes by hand (I mean, not entirely by hand, but
you get the idea). It was daunting at first but is getting easier and easier
all the time, and we have a better handle on how things work as a result.
What's going on under the hood, what the mass of command line parameters
actually mean, and so on.

The documentation is lacking. Badly. And doesn't seem to keep pace with the
features that are added in every point release, or in the alpha/beta versions.
A lot of what we've learned has been by experimentation, poking at things,
following the k8s user discussions, watching Github, reading code.

~~~
mdaniel
Can you elaborate on the information you wish was documented but isn't?

Also, have you seen the box at the bottom of the docs that says "I wish this
page ..."? It goes right into their issue tracker, which increases the
likelihood of something getting fixed.

~~~
jat850
To be fair, I have not seen or used that link, but I will take note of it for
the future - thank you, that's very helpful.

I should keep notes on specifics, so I apologize that I can't highlight a
particular thing that I've been frustrated by in a given moment. I'll
certainly say that the documentation is improving.

As a general item, I think my biggest struggle has been hitting a wall with
the documentation - there are some things that are left almost as exercises to
the reader, especially getting into more advanced topics (how does one do
rolling upgrades or canary deployments in more esoteric situations, how might
one do load balancing properly in a master-slave style configuration versus
something more round-robin oriented, etc.)

And I don't like to levy a complaint without acknowledging that anything
opensource is ripe for improvement by contribution, but my professional
situation prevents me from doing so. Trust that I would love nothing more than
to help and not just make an empty observation or moan about it.

~~~
mdaniel
So I typed out this whole answer and only realized at the end that I should
have asked: have you tried their Slack channel:
[https://kubernetes.slack.com/messages/kubernetes-
users/](https://kubernetes.slack.com/messages/kubernetes-users/) and/or their
mailing list [https://groups.google.com/forum/#!forum/kubernetes-
users](https://groups.google.com/forum/#!forum/kubernetes-users) for getting
answers to your specific concerns?

 _how does one do rolling upgrades or canary deployments in more esoteric
situations_

So there are a couple of moving parts to that: foremost, Kubernetes is only a
tool, and so I would doubt there is The One True Way(tm) of doing canary
deployments -- and I'm taking liberties with your comment in that I presume
you don't mean the literal rolling-upgrade which is ``kubectl rolling-update``
and is _very smart_. Having said "it's just a tool," the first thing that
sprung to mind when reading canary deployments was the health checker built
into the ReplicationControllers. There are several ways one can check on the
"health" of a Pod, and k8s will remove any Pod from service that has declared
itself ineligible for service.

If the QoS metrics are too broad for any one given Pod to have insight into,
k8s has a very robust API through which external actors can manipulate the
state of the cluster, rolling back the upgrade if it turns out the new deploy
is proving problematic.

I hope he doesn't throw tomatoes at me for suggesting this, but Kelsey
Hightower
<[https://github.com/kelseyhightower>](https://github.com/kelseyhightower>) is
an amazing, absolutely amazing, community resource. If you don't find
inspiration from the talks he has given, then reach out on Twitter and ask for
a good place to read up on your concerns. I bet if he doesn't know the answer,
he will know the next contact for you to try.

 _load balancing properly in a master-slave style configuration versus
something more round-robin oriented_

Now there is something you and k8s may genuinely have a disagreement about -
in its mental model (to the best of my knowledge) every Pod that matches the
selector for a Service is a candidate to receive traffic. So in that way,
there is no "slave" because everyone is equal. However, if I am hearing the
master-slave question correctly, failover is automagic because of the
aforementioned health check.

~~~
jat850
I have, yes, and there's no doubt they're both crucial in getting answers to
some of those more advanced topics.

 _edit_ I want to also add that the Kubernetes community meeting is fantastic.
I don't attend every week but I do play catchup on the videos released after.

Regarding rolling updates, I have found that looking at the kubernetes API -
and discerning what kubectl is doing under the hood - is helpful in some
cases. We've taken to API inspection to assist with some of the more complex
update patterns we're trying to address.

Completely agreed on the value of Kelsey to the community. I hesitate about
contacting any one person directly on a topic, but his guides and github repo
are just outstanding.

On the load balancing/master slave thing - I would have agreed completely with
you. Master/slave configurations seem antithetical to some of the core
Kubernetes concepts... but then the waters are becoming muddied because it
seems like PetSets are a response to that missing pattern. I think you're
right on the mental model. Every pod is (from what I understand), or at least
should be, seen as an equal candidate, the only thing being "can it receive
and service requests, if yes, kubernetes doesn't care".

Failover by letting Kubernetes do the dirty work is, of course, an option. If
the master or a slave dies - it'll come back up. Except... it's also at odds
with anything that handles its own clustering model (thinking Redis Sentinel
or Redis Cluster, or InfluxDB when it had clustering support). Sometimes
"coming back up" in a system that has an orthogonal view to Kubernetes is
challenging.

It also doesn't accommodate for the situation where... what if Kubernetes
_doesn 't_ bring it back up for some reason? If the pod was killed for
resource contention, or something along those lines. And now I have two
slaves, no master, nowhere to feed writes?

I dont't complain too loudly about these things, because most often the
solution is (correctly) to think differently about it, but there are real-
world considerations that don't always fit perfectly with the existing
Kubernetes model - but we may still have to use them from time to time.

~~~
smarterclayton
PetSet co-author here:

Think of PetSet as a design pattern for running a cluster on Kubernetes. It
should offer a convenient pattern that someone can use to Do-The-Right-Thing.

The hard part of clustering software is configuration change management. If an
admin adds a new member to the cluster by hand, they're usually correct (the
human "understands" that the change is safe because no partition is currently
going on). But an admin can also induce split brain. PetSets on Kube try to
make configuration changes predictable by leveraging their own strong
consistency (backed by the etcd quorum underneath the Kube masters) to tell
each member the correct configuration at a point in time.

The PetSet goal is to allow relative experts in a particular technology to
make a set of safe decisions for their particular clustered software that
behaves _exactly_ as they would predict, so that they can make a set of
recommendations about how to deploy on the platform.

For instance, CoreOS is working to provide a set of recommendations for etcd
on pet sets that would be correct for anyone who wants to run a cluster
safely, and encode that into tutorials / docs / actual pet definitions.
They're also driving requirements. Other folks have been doing that for
elastic search, zookeeper, galera, cassandra, etc. PetSets won't be "done"
until it's possible to take the real world considerations into account safely
when writing one.

~~~
jat850
That's really valuable insight, thank you. The biggest struggle so far has
been with zookeeper and Kafka. Things that are stateful have posed a lot of
difficulty - from a mental perspective more than fighting against Kubernetes
specifically, just trying to think and adhere more to microservices
principles.

I'm following PetSets very closely and I think they're going to help a great
deal.

Is it accurate that PetSets were introduced to compensate for the type of
thing I've singled out? That certain models just don't quite fit with the
"everything is equal and interchangeable" notion? Or does it just feel that
way? I don't want to end up missing the point of PetSets or using them for
something they're not truly intended for, or leaning on them only to hit a
stumbling block.

~~~
smarterclayton
Yes. PetSets are intended to be step one of "some things need to be stable and
replaceable at the same time". So that unit can then be composed into apps
(i.e. Zookeeper + Kafka might be a PetSet for ZK and multiple scale groups for
Kafka). basically trying to boil the problem space down into "what do cluster
aware software need to be safe", like protection from split brain and
predictable ordering.

There is likely to be more needed (on top or around) because we need to also
solve problems for highly available "leaders" and active/passive setups. Think
of what something like pacemaker does to ensure you can run two Linux systems
in an HA fashion, and then map the necessary parts to Kube (fencing, ip
failover, cloning, failure detection). Still a lot of hard problems to solve -
PetSets are the first building block.

------
nzoschke
These articles never mention the elephant in the room, AWS.

How many containers are running on Elastic Beanstalk and ECS?

I'd wager magnitudes more are running containers by reading the docs and
clicking around than those that mastering the cutting edge challenges getting
running on Mesos, Kube or Swarm.

Another blind spot is Heroku. Every day new businesses spin up containers with
a 'git push Heroku master' and don't even know about it.

All these providers, platforms and tools have their place.

I simply don't think the "winning the war" talk is accurate or constructive.

Disclaimer: I worked on the container systems at Heroku and now manage 100s of
production ECS clusters at Convox.

~~~
colemickens
What "cutting edge challenges" are there to get running on Mesos/Kube/Swarm?
Are you just referring to getting the cluster booted? There are just so many
options for that these days.

Just curious, I was skimming through ECS docs last night and it seems like
some of the basic building blocks look very similar to their equivalent
constructs in Kube/Nomad.

~~~
nzoschke
Automating VMs with Elastic Beanstalk and deploying to a platform like Heroku
are effectively solved problems.

Container orchestration brings in new challenges like operating a consensus
database, managing data volumes and load balancing.

And this is on top of VM management.

Yes, ECS, Kube, Nomad are very similar. All relatively hard to use right now
compared to their simpler ancestors.

Here is more in depth info about the challenges of ECS:

[https://convox.com/blog/ecs-challenges/](https://convox.com/blog/ecs-
challenges/)

------
runako
On Safari OS X, the article is blocked by a full-page ad that I can't dismiss.

~~~
the_duke
Don't see any ads on Chrome with Adblock Plus.

It shows 32 blocked requests though (ouch).

~~~
cptskippy
That seems light for most pages...

~~~
the_duke
About average, I'd say.

Quick check:

* cnn.com: 30-40 * bbc.com: 10-15 * washingtonpost.com: 10-25 * medium.com: 1-2 * teccrunch.com: 25-40

------
kbredemeier
Innovation is what makes this industry so exciting to be a part of. I joined
the tech world, via Holberton School[1] just a mere nine months ago, and
already so much has changed. It makes it challenging to keep up, but that's
the fun part.

The future of technology is held in the hands of open source projects.

[1] [https://www.holbertonschool.com](https://www.holbertonschool.com)

