
Mesos Borgs Google’s Kubernetes Right Back - rbanffy
https://www.nextplatform.com/2017/09/07/mesos-borgs-googles-kubernetes-right-back/
======
cheap
I'm not one to write negative reviews of open source tech. Typically, everyone
has skin in the game for one reason or the other, and diversity in tech is
positively great.

But Mesosphere DC/OS is purely advertising and marketing driven.

Their "Docker support" simply means they use normal Mesos worker processes to
shell out to the Docker CLI across a cluster. They tightly wrap Hashicorp
Vault and label it their own solution. Marathon has terrible support for
security and application deployments geared toward enterprise teams. Hell,
deployments can write over each other's network volumes and setting IAM roles
can be sniffed straight out of unencrypted HTTPS headers. Don't get me started
with Minuteman, Mesos DNS, meshing IPTables rules, and the hundreds of hacks
around missing IP-per-container/network virtualization that even Solaris has
had for the past 20 years (Crossbow anyone?).

The only thing people want right now is AWS in private/hybrid cloud. All the
big movers are getting off AWS. If not, they're either too small to matter or
are positioning their "cloud partnership" as a buy-out to Ma'Amazon.

~~~
mgummelt
Almost all of this is out of date. Did you try DC/OS a long time ago, maybe?

> Their "Docker support" simply means they use normal Mesos worker processes
> to shell out to the Docker CLI across a cluster.

This hasn't been true for quite some time now:
[http://mesos.apache.org/documentation/latest/container-
image...](http://mesos.apache.org/documentation/latest/container-image/)

> hundreds of hacks around missing IP-per-container/network virtualization

DC/OS does have network virtualization and IP-per-container:
[https://dcos.io/docs/1.9/networking/virtual-networks/ip-
per-...](https://dcos.io/docs/1.9/networking/virtual-networks/ip-per-
container/)

Also, what's an unencrypted HTTPS header?

~~~
irishasaurus
Oh man there are decrypted HTTPS headers? I can stop using Wireshark then!

[https://jimshaver.net/2015/02/11/decrypting-tls-browser-
traf...](https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-
wireshark-the-easy-way/)

~~~
wpietri
Holy moly, that is a helpful link. I had been doing it the old-fashioned way
that used the server key. It's so great that browsers added support for
logging session keys.

------
sandGorgon
Spark on Kubernetes (which is still not fully production ready) will be the
clearest indicator of k8s taking over the space occupied by Mesos.

Mesos/Yarn is used by spark to schedule batch jobs in a distributed manner on
a spark cluster. So they have to have very deep knowledge of the resource
availability and the needs of the jobs being scheduled. We are not talking
static resource allocation like a container, but scheduling a particular piece
of processing in real time to one of the nodes managed by Mesos/Yarn.

Kubernetes does not come with this level of awareness today. Mesos was always
built with this in mind. So for Mesos, the scheduling of containers is a
subset of its functionality.

However kubernetes is getting there -
[https://github.com/kubernetes/kubernetes/issues/34377](https://github.com/kubernetes/kubernetes/issues/34377)
[https://github.com/apache-spark-
on-k8s/spark/issues/4](https://github.com/apache-spark-on-k8s/spark/issues/4)
[https://issues.apache.org/jira/plugins/servlet/mobile#issue/...](https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-18278)

Today if you want to deploy a fleet of containers and schedule a bunch of
processes across them in a resource aware manner...you have to use both
kubernetes and mesos.

------
wmf
In general, nested orchestration (k8s on Mesos, OpenStack on k8s, Docker on
VMware, etc.) is an infrastructure smell due to the significant overlap and
impedance mismatch between the systems.

------
exelius
Dunno this is really a huge deal -- they're different tools with different
purposes.

I've always viewed Mesos as a bare-metal orchestration platform. From there,
you can deploy k8s, VMware, Linux KVM, whatever to the nodes under management
by Mesos.

k8s is more of a container management platform that can be used for
applications to tie arbitrary software architectures into a set of unified
operations tooling.

So you may host your cluster on k8s, but you would use Mesos to auto-scale the
active servers in your clusters for say, power management. But Mesos is
something you probably won't have a need for until you're well past managing
one k8s cluster.

~~~
pram
Does kubernetes actually work as a Mesos framework? I thought that was the
point of marathon.

~~~
nemothekid
I don't think k8s operates as a Mesos framework, but the use cases for k8s is
a lot closer to Marathon than Mesos. For most use cases k8s and marathon
satisfy the same needs. If the only framework you are running on Mesos is
Marathon, then you may be better off with k8s.

~~~
SEJeff
"I don't think k8s operates as a Meoss framework"

Have you ever used Mesos? It is the only way k8s _could_ run ontop of Mesos.
Also:

[https://github.com/kubernetes-incubator/kube-mesos-
framework](https://github.com/kubernetes-incubator/kube-mesos-framework)

Also, you could maybe compare Marathon to Kubernetes Dashboard + the
Kubernetes apiserver _maybe_ , but comparing the two directly is not remotely
close.

With Mesos + Marathon, you still need a load balancer for both North <\--->
South AND East <\---> West cluster traffice. This is builtin to Kubernetes via
Service and Ingress objects and their associated controllers. There is also
Federation, and so much more.

Mesos is just a resource scheduler, and a very good two level one at that,
built to prove the theory of two level schedulers working well with
interactive and batch workloads concurrently. Kubernetes is the entire package
and includes a default scheduler, but allows you to plug in your own.

Disclaimer: I spent a full year working on Mesos and went to MesosCon twice.
After speaking in person with Kelsey Hightower at Monitorama, he convinced me
to switch to Kubernetes on purely technical reasons. The stuff that you have
to DIY on Mesos is built sensibly and pluggably in Kubernetes. From a cluster
operator standpoint frankly, Kubernetes is leagues ahead.

Lots of fortune 500s and enterprise customers find the lack if sensible RBAC
and real to the core auditing support in Mesos to be seriously lacking.

[https://twitter.com/nirajtolia/status/903437346443378688](https://twitter.com/nirajtolia/status/903437346443378688)

------
elvinyung
"You will be assimilated. Resistance is futile."

------
unlogic
r/titlegore

