
Kubernetes V1 Released - Oletros
http://googlecloudplatform.blogspot.com/2015/07/Kubernetes-V1-Released.html
======
StevePerkins
Can anyone recommend a good "Consultant's", "Solutions Architect's", or "Top-
Right Quadrant on a Silly Gardner Chart" overview of how Kubernetes competes
or cooperates with Mesos? It feels like there's quite a bit of dense
conceptual reading you have to plow through before you can even start to talk
about what these things do.

~~~
bkeroack
To add to the others, in a nutshell:

\- Mesos is a generalized, low level framework for distributing workloads over
multiple nodes. It provides mechanism not policy. Therefore it requires quite
a bit of up-front work to build something usable for any given application.

\- Kubernetes is an opinionated cluster execution tool. It provides tools and
a curated workflow for running distributed containerized applications. It's
generally pretty quick and easy to get running.

\- Mesos has a rich, resource-aware task scheduler. You can specify that your
application requires X CPU units and Y RAM units and it will find the optimum
node to run the task on.

\- By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's
no way to specify the expected resource utilization for pods, and the
scheduler simply tries to spread out replicas as much as possible throughout
the available nodes.

People are (rightly) excited about things like Mesosphere which could allow
the best of both worlds: the ease and API of Kubernetes with a powerful Mesos
resource scheduler, not to mention nice-to-haves like a Web UI with pretty
visualizations.

You can now cut me a check for 50% of the consulting revenue you get from this
information. :)

1\. The scheduler is intentionally simple and pluggable, to allow improvements
easily in the future. My statements only apply to the current state of
Kubernetes as deployed today.

~~~
davidooo
The Kubernetes scheduler also does resource-aware scheduling. You're correct
that it tries to spread replicas across nodes, but it only spreads them across
the nodes that have enough free resources for the container (more precisely,
Pod) that it's scheduling.

~~~
bkeroack
Nowhere in the pod spec is there a way to specify resource constraints or even
hints
([https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897)).

So even if the scheduler is vaguely resource-aware (I'm not convinced that's
true) it would be entirely static, based on things like container count.

~~~
davidooo
Currently resource requirements are specified only on containers, not Pods.
The requirements for the Pod are computed by adding up the requirements of the
containers within the Pod.

To be more concrete: Within the PodSpec type that you linked to, there is a
field of type []Container. Within the Container type there is a field called
Resources which is of type ResourceRequirements. ResourceRequirements lets you
specify resource requirements of the container. The resource requirements of
the Pod are computed by adding up the resource requirements of the containers
that run within the Pod.

In addition to resource-based scheduling, we also support "label selectors"
which allows you to label nodes with key/value pairs and then say that a Pod
should only run on nodes with particular labels. That's specified in the
NodeSelector field of the PodSpec (which you linked to).

~~~
bkeroack
Fair enough! :) I overlooked that aspect of the Container type, obviously.

------
jcastro
If you're using Ubuntu we've got a bundle of Juju charms for those who want to
get up and running:

\- [http://insights.ubuntu.com/2015/07/21/introducing-
kubernetes...](http://insights.ubuntu.com/2015/07/21/introducing-kubernetes-
version-1-0/)

\- [https://jujucharms.com/u/kubernetes/kubernetes-
cluster](https://jujucharms.com/u/kubernetes/kubernetes-cluster)

PRs and comments welcome!

------
nickbauman
I've found that Kubernetes is a big hammer. If your problem can be backed by a
web app, you should start with AppEngine. If you need specialty library
support or wider computational space per instance, you can move your AppEngine
app to ManagedVMs (which uses Kubernetes under the covers). If you need
special, "grid-like" services where you need exquisite control over the entire
stack, only then does it make sense to use raw Kubernetes and Docker. And you
will spend a lot of time getting it right.

~~~
puja108
ManagedVMs don't use k8s under the covers afaik. However there's GKE (Google
Container Engine), which goes on top of (at least 3) MVMs and that one does
use k8s. While it is true that going with a PaaS(-like) solution like
AppEngine or Heroku is easy in the beginning it can get pretty expensive
pretty fast and it limits you in the choice of languages, frameworks, and data
stores you can use. This can in some instances bring technical debt with it
that will pose a hurdle when growing. Actually, using Docker combined with an
orchestration layer like k8s is supposed to give the the ease-of-use of PaaS
with the flexibility of IaaS (or MVMs), however, managing sth like k8s by
yourself is not that easy and you will need quite a bunch of other tools on
top, i.e. for monitoring and stuff, which paves the way for Container-as-a-
Service solutions, like RancherOS, tutum, or Giant Swarm (disclaimer, I'm part
of the latter company)

~~~
dlor
Correct, MVMs does not use Kubernetes today. We're looking at rebasing onto
GKE now that they've hit 1.0, but nothing is set in stone yet.

Disclaimer: I'm a tech lead on Managed VMs/App Engine at Google

~~~
alooPotato
Since you're already here, any chance you can talk a little bit more about
upcoming plans for managed vms? I hear there is a lot of investment in that
area, but what will that mean technically going forward? How are things going
to change so we can plan better?

~~~
dlor
I don't want to hijack the Kubernetes launch thread, but feel free to send me
an email at dlorenc@google.com and I'll answer whatever I can. We have a lot
of exciting stuff going out soon.

------
geku
Getting started with Kubernetes is pretty easy and I wrote a guide to quickly
run it on your local machine: [https://www.cloudgear.net/blog/2015/5-minutes-
kubernetes-set...](https://www.cloudgear.net/blog/2015/5-minutes-kubernetes-
setup/)

It's not up to date with Kubernetes 1.0.0 but I'll update the images as soon
as the final version 1 is tagged.

~~~
TheIronYuppie
I definitely don't want this to come off as a sales pitch, but you can get
started in one click using Google Container Engine (and $300 in free credit)
as well.

Full disclosure: I work at Google on Kubernetes

~~~
gtaylor
We were looking at this, but noticed that you have to one run cluster per
availability zone. Any plans for being able to run a cluster across an entire
region within GCE?

~~~
alex-mohr
Yes, we've heard from a number of people who want that and will improve
regional support.

Current ideas are either a single regional cluster or via federation of
multiple zonal clusters.

See eg
[https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/proposals/federation.md)
for an proposal on the latter.

~~~
gtaylor
FWIW, I'd love to have Kubernetes clusters spanning a region with multiple
regions/providers managed by Ubernetes. That would be the sweet spot for our
particular usage case.

This is only one point of data for you, of course.

------
rjusher
I might come across as ignorant but what is the relationship between
Kubernetes and Docker, because when I was reading the article I tought of it
as a Docker competitor, but further down in the comments, there is one that
says they do different jobs.

And that confused me.

~~~
TheIronYuppie
Docker and Kubernetes works hand in hand. That is to say, if you choose Docker
as your container format, Kubernetes runs Docker on every node to run your
container. Kubernetes focuses on _what_ Docker should run, and how to move
those container workloads around.

Docker also has Docker Swarm, which can be thought of as a competitor in some
ways. But Google will be a heavy supporter of their container format for a
long time to come.

Full Disclosure: I work at Google on Kubernetes

~~~
rjusher
So Kubernetes compliments Docker, but how it complements it.

I had tested Docker just for fun, thinking that maybe I could implement it in
the way I work, and sure it is a super tool for developing (far better than
Virtual Machines), but deploying was kind of nightmerish, for what I
understood Docker wasn't at the time ready for being a deployment tool.

Does Kubernetes fixes or extends Docker in this way

~~~
bkeroack
Think of them as different layers. If you're a front end web dev, it's sort of
like SASS vs CSS: the former is a layer on top of the latter that makes it
more powerful/convenient/easier to use.

At the bottom of the stack (most low level) is the Docker runtime. It knows
how to run containers on the local machine. It can link them together, manage
volumes, etc but at the core it is a _single machine_ system. (That's probably
why Docker, Inc has developed their proprietary orchestration tools like
swarm).

Layered on top of that are container-native OSes like CoreOS. CoreOS provides
facilities for running distributed containers on multiple physical nodes. It
handles things like replication and restarting failed containers (actually
fleet "units"). This is a huge improvement over vanilla Docker, but it's still
pretty low level. If you want to run a real production application with
dependencies it can be tedious. For example, linking together containers that
run on different nodes. How does container A find container B (which may be
running on any one of N nodes)? To solve this you have to do things like the
Ambassador Pattern[1]. Any complex application deployment involves essentially
building discovery and dependency management from scratch.

Layered on top of this is Kubernetes (it runs on CoreOS but also Ubuntu and
others). As said elsewhere in this post, k8s provides an opinionated workflow
that allows you to build distributed application deployments without the pain
of implementing things like low-level discovery. You describe your application
in terms of containers, ports, and services and k8s takes care of spawning
them, managing replica count (restarting/migrating if necessary) and discovery
(via DNS or environment variables).

One of the very convenient things about k8s (unlike vanilla Docker) is that
all containers within a pod can find each other via localhost, so you don't
have to maintain tedious webs of container links. In general it takes the
world of containerization from "Cool technology, but good luck migrating
production over" to "I think we could do this!".

1\. [https://coreos.com/blog/docker-dynamic-ambassador-powered-
by...](https://coreos.com/blog/docker-dynamic-ambassador-powered-by-etcd/)

~~~
TheIronYuppie
was about to write a response, but bkeroack did a perfect job with the above
:)

Full disclosure: I work at Google on Kubernetes

------
Oletros
And related to that, CoreOS has launcehd a preview of its Kubernetes comercial
platform [0]

[0] [http://techcrunch.com/2015/07/21/coreos-launches-preview-
of-...](http://techcrunch.com/2015/07/21/coreos-launches-preview-of-tectonic-
its-commercial-kubernetes-platform/?ncid=rss)

------
kordless
Wercker has a post on continous integration for containers to Kubernetes which
Micha, their CEO, did back in June:
[http://blog.wercker.com/2015/06/23/Deploying-minimal-
contain...](http://blog.wercker.com/2015/06/23/Deploying-minimal-containers-
to-kubernetes.html), which uses Google's Container Registry for pushing
images.

If anyone is interested, I just wrapped up a similar post for deploying
containers to Giant Swarm from Wercker, no Docker required: I just got done
doing a continuos integration post for containers using Wercker and Giant
Swarm: [https://github.com/giantswarm/swarm-
wercker](https://github.com/giantswarm/swarm-wercker).

------
arianvanp
I'm hoping for ACI support soon so that I can use rkt instead of docker. :-)

~~~
TheIronYuppie
I have good news for you :)

[https://blog.kismatic.com/running-rkt-on-
kubernetes/](https://blog.kismatic.com/running-rkt-on-kubernetes/)

Full Disclosure: I work at Google on Kubernetes

~~~
arianvanp
Yes you can use rkt for docker images, but I want to be able to use ACI's.
which doesn't seem possible yet[0].

But awesome that there's at least some support! :D

[0] -
[https://github.com/GoogleCloudPlatform/kubernetes/issues/720...](https://github.com/GoogleCloudPlatform/kubernetes/issues/7203)

------
smegel
> containers scheduled < 5s on average

That seems awfully slow for a fancy chroot. I use KVM to bring up WinXP
snapshot VMs in around 2s to a running state...maybe they mean 5ms?

~~~
cjcullen
5s is the right number, but it leaves out that it is 99%ile, on a 100-node,
3000-pod cluster. See
[https://github.com/GoogleCloudPlatform/kubernetes/issues/395...](https://github.com/GoogleCloudPlatform/kubernetes/issues/3954).

------
jacques_chester
Interesting to see IBM placing a bet each way by joining the new foundation,
seeing as they're already in the Cloud Foundry Foundation.

Edit: and I see the Cloud Foundry Foundation logo on the Cloud Native
Foundation homepage. It's Foundations all the way down.

(Disclaimer: I work for another CFF member, Pivotal).

~~~
justincormack
Its pretty orthogonal to cloud foundry, which is a composite infrastructure,
while these are lower level products that can be used to build PaaS.

~~~
jacques_chester
Right -- the analogous component is Diego.

I guess I got caught up by the inside baseball.

------
alanh
Is Kubernetes pronounced "koo bur NET ease" or am I way off?

~~~
a-robinson
Yes, that's how it's pronounced.

------
wereHamster
What's with the recent adoption of the .io TLD @ google? cncf.io, gcr.io, ...

~~~
dguaraglia
I guess it's driven by the same motivation everyone else has: it's impossible
to get a decent .com domain.

------
barkingcat
how do you pronounce that word?

~~~
packetslave
coo-bur-net-eees

