
Run Kubernetes on top of DC/OS - tobilg
https://mesosphere.com/blog/kubernetes-dcos/
======
slap_shot
I recently reached out to Mesosphere to better understand the value and
options available to my company if we switched to Mesos (and thus Mesosphere).
We currently run Kubernetes across all major cloud providers and we'd like
some help.

I was startled by how dogmatic they are about their pricing model: per node,
annually. Our entire business model is running dynamically scaling data
pipelines for companies. We spin nodes up and down programmatically based on a
pipeline's history and a dozen other factors.

I had a lot of back and forth with them and eventually their CFO and they just
could not understand the concept of a "node hour" or anything of the like.
Eventually, it was determined if we paid the per-node annual price for our
"average" (completely different story, but they calculated this about as wrong
as you could have) usage and we went our separate ways.

The product seems really cool but as someone who runs a company that easily
spin up thousands of nodes in the middle of the night for (for a grand total
of $40/hr on GKE...) this pricing model just seems antiquated. I imagine that
all companies will be driving their usage to this practice in due time. I
really hope to see the industry change.

~~~
rectang
_> Mesos (and thus Mesosphere)_

Mesos is an open source project. Would you consider any other vendors besides
Mesosphere?

~~~
don71
DC/OS is also open source. What are your concerns with it?

~~~
trengrj
Components aren't open source. Still waiting for the new hdfs package to be
released see [https://github.com/mesosphere/hdfs-
deprecated/blob/master/RE...](https://github.com/mesosphere/hdfs-
deprecated/blob/master/README.md)

~~~
tknaup
The new package is here: [http://universe.dcos.io/#/package/beta-
hdfs/version/latest](http://universe.dcos.io/#/package/beta-
hdfs/version/latest)

~~~
trengrj
Yes the package is available but as far as I can see not open source yet.

~~~
spahl
Here it is: [https://github.com/mesosphere/dcos-
commons/tree/master/frame...](https://github.com/mesosphere/dcos-
commons/tree/master/frameworks/hdfs)

~~~
trengrj
Excellent. Thanks!

------
bogomipz
I was confused by this passage:

>"Running Kubernetes on DC/OS allows you to run different types of workloads
(more explicitly, both the stateless and stateful components that make up most
modern applications) on the same infrastructure."

Can someoene answer - how does Running Kubernetes on top of DC/OS help you run
stateful apps on Kubernetes?

Or is the meaning that DC/OS is better for running stateful services and then
you can use K8 to run your stateless services?

Its been a while since I've used Mesos, is the path for running stateful
services on complete and very compelling now?

Thanks.

~~~
estill01
Currently running stateful apps on a DC/OS cluster + HA DBs. Fairly straight
forward to get stateful working now + there are libs to migrate data around to
chase your apps/services when/if Mesos/Marathon relocates them (e.g. after a
service restart/crash) if you need your data co-located with a managed
app/service.

The thing I'm most interested in exploring with K8+DC/OS is having DC/OS
manage a couple of K8 instances so we can isolate 'virtual' clusters for
various envs/apps. I suppose you could do this already with Marathon (the
DC/OS built-in container manager), but we're not. Psyched to benefit from the
K8 community + have the underlying DC/OS VM control/management plane.

~~~
bogomipz
Thanks, can you elaborate on the libs available to "migrate your data around"?
Does Mesosphere reschedule your DB to another node that has an equivalent
persistent and reserved storage(SSD etc.) volume configured on it?

Are you using Portworx for this?

~~~
estill01
> "can you elaborate on the libs available to 'migrate your data around'? ...
> Are you using Portworx for this?"

The tool I had in mind was 'REX-
Ray'[[https://mesosphere.github.io/marathon/docs/external-
volumes....](https://mesosphere.github.io/marathon/docs/external-
volumes.html#specifying-an-external-volume)].

That said, we're not actually doing the 'chasing db' config. Instead we run a
HA Neo4j DB deployment as a Marathon service pegged to a handful of nodes each
with local persistent volumes allocated to Neo4j. I.e. we can allocate a % of
a node's resources to 'static' Neo4j deploys, and then let Marathon
dynamically manage any remaining free resources on the nodes. Our other
services then use the Marathon service DNS to look up the Neo4j service for
read/write.

Portworx looks cool too -- will need to investigate.

Also, the DC/OS documentation is quite good in general if you're looking to
dig in on this:
[https://dcos.io/docs/1.9/storage/](https://dcos.io/docs/1.9/storage/)

> "Does Mesosphere reschedule your DB to another node that has an equivalent
> persistent and reserved storage(SSD etc.) volume configured on it?"

Yes/it can, but in that config you're booting up new/empty storage volumes.
Obviously not what you want for many core persistence requirements though
great for caches. We'll probably opt for this config near-term for our web-
server SSR cache.

~~~
ferrantim
Michael from Portworx here. Thanks for the shout out. Just for some context,
we just announced a partnership with Mesosphere today to help accelerate
adoption of DCOS for stateful services [0] in fact. We handle the automation
of all the state management mentioned above, not just volume provisioning. Our
customers include big companies like GE and Dreamworks but also a lot of
smaller companies. You can use PX-Dev[1] for free up to 3 nodes. Would love
feedback.

[0] [http://m.marketwired.com/press-release/mesosphere-
partners-w...](http://m.marketwired.com/press-release/mesosphere-partners-
with-portworx-accelerate-production-deployments-dc-os-enterprise-2232659.htm)

[1] [https://docs.portworx.com](https://docs.portworx.com)

~~~
estill01
Cool -- We're a small operation atm so will take a look at that dev tier. For
more context, we also do some block storage off-cluster in GCE.

------
devrandomguy
Could anyone point out the boundary between the open-source Mesos / Marathon
stack, and the commercial DC/OS? The official documentation
([https://docs.mesosphere.com/1.9/overview/what-is-
dcos/](https://docs.mesosphere.com/1.9/overview/what-is-dcos/)) doesn't make
it clear. What capabilities does DC/OS have that are not available in the
open-source portion of the stack?

> As a distributed system, DC/OS includes a group of agent nodes that are
> coordinated by a group of master nodes. Like other distributed systems,
> several of the components running on the master nodes perform leader
> election with their peers.

This just sounds like vanilla Mesos masters and slaves. DC/OS "runs on top of"
that, but what is it actually doing? Is the DC/OS another service that just
enables the Mesos masters to initially discover and replace each other? Could
it not be replaced with Zookeeper or Consul? That seems like a small piece of
the puzzle, what would make it the one expensive thing, while the rest of the
system is free? Is the overall stack actually shareware, rather than a
community of independent open-source services?

The reason I am so curious, is because after running a small demo a year ago,
the Mesos stack looked really promising. The only thing holding me back from
proposing a large scale trial, for comparison to our sprawling Heroku/ECS/EBS
setup, was the feeling that I was not understanding a crucial part of the
architecture, and not understanding the pricing, if that is even applicable
(couldn't find price info anywhere! How do I quantify that part?)

~~~
nemothekid
As I understand it, DC/OS is essentially a nicer interface for Mesos /
Marathon with baked in monitoring, service discovery, security add-ons and a
"package" manager-esque framework manager.

DC/OS is actually open source (I think) so I don't think it has any
capabilities you could get for free.

------
bogomipz
>"Kubernetes on top of Mesos through DC/OS more closely matches Google’s own
architecture; where Kubernetes is a service running within VMs that are
managed by Google’s proprietary Borg platform."

My understanding was that Google runs containers in a VM for security. Mesos
uses the Docker container executor not a VM. How does this more closely
resemble Google's Borg/VM model?

~~~
tknaup
Mesosphere co-founder here.

You're correct in that GCP runs k8s in VMs, DC/OS doesn't. What's similar is
that there's a resource manager underneath - Borg for GCP, Mesos for DC/OS.
They serve similar purposes like resource management, isolation, and operating
the services on top.

~~~
bogomipz
>"What's similar is that there's a resource manager underneath - Borg for GCP,
Mesos for DC/OS."

Maybe I don't fully understand DC/OS then. I was under the impression that
DC/OS was simply a distro for Mesosphere. But your comment make me think that
either my understanding is incorrect or else DC/OS has become something more
than a Mesos distro. Could you elaborate? Thanks.

------
filereaper
Excellent, was asking about this a while ago [1], looking forward to trying
this out on Azure.

[1]
[https://news.ycombinator.com/item?id=14907878](https://news.ycombinator.com/item?id=14907878)

