
CoreOS (YC S13) Raises $12M to Bring Kubernetes to the Enterprise - craneca0
http://techcrunch.com/2015/04/06/coreos-raises-12m-funding-round-led-by-google-ventures-to-bring-kubernetes-to-the-enterprise/
======
raspasov
Maybe it's just me, but I fail to see how all those project make the
developer/devops person's life less complex. I don't have the experience to
have managed a cluster of 1000s of machines but I HAVE managed a cluster of
~50 nodes with MySQL, ElasticSearch, RabbitMQ, ZeroMQ, PHP, Clojure and the
whole gang at various points in time. I am yet to see a single project (Mesos,
Kubernetes, Docker, whatever) that would DRAMATICALLY make my life so much
better. Not trying to be overly negative here, just looking for answers and
better solutions.

EDIT I also continuously wonder how do things like Ansible fit into the
picture. Are they competitors? Are they supplemental to the likes of
Mesos/Kubernetes? Are they orthogonal?

~~~
brendandburns
Disclaimer: lead engineer on kubernetes here...

It really depends on how you do deployment. Containers provide deployment (and
more important rollback) that is better than other deployment tools like
Puppet/Chef/... because they are atomic (they either work, or they fail, they
don't get stuck in the middle) and they package up all of their dependencies
within them, so that they don't have the "well it worked on my machine"
problems.

Systems like Mesos and Kuberenetes, decouple applications from the individual
machines (and the operating system on those machines), and are online systems
with self-healing properties (so that they will fix themselves rather than
waking you up in the middle of the night)

k8s and mesos turn containers into an API that spans an entire fleet of
machines, and enables you to dynamically use (and re-use) the fleet of
machines for multiple different applications. No more dedicated boxes for
mysql, mongo, etc. This in turn enables you to have an easier ops experience,
because every single machine in your fleet is homogenous (same OS, same
patches, etc) OS management is abstracted away from Application management, so
that they don't interfere with each other. Since things in the API are
expressed in terms of applications, it's easy to add health checks and
automatic restart to the system, and provide self-healing properties as well.
Both kubernetes and Mesos also make replication a first-order primitive so
that it is easy to scale in response to load.

Ansible is sort of orthogonal to systems like Kubernetes and Mesos. Kubernetes
and Mesos are designed to be online, self-repairing systems. Ansible is a way
to easily execute commands on a bunch of machines. I can see collaborative use
cases, where you generally use Kubernetes for deployments, but use Ansible for
querying some data while debugging, or somesuch.

Anyway, sorry for the extended response. There actually is way more that I
could say about the topic ;)

~~~
raspasov
Thanks for the detailed response. The point about atomic deploys is a good
one. However, if we all agree that immutable deployments are a good thing,
I've been wondering how is that fundamentally different from launching new
instances via AWS/Rackspace/Google API? Is there really a fundamental
difference between shipping a Docker container to a set of servers, or just
relaunching/rebuilding servers?

You also mention the "no dedicated Mysql, Mongo boxes" etc. I am 100% for
that. However, how can you really make that work with databases/systems like
MySQL which were fundamentally designed to work on one machine, and scaling
them across machines is usually very painful, or at least, let's say, not very
"idiomatic" (if I can use that word here lol). I can see the auto scaling part
working with distributed databases like Riak/Cassandra, but even there the
solution is not clear-cut and "out of the box". It still feels like some
manual orchestration work is needed - correct me if I'm wrong.

I can totally see the "online, self-repairing point" but only for application
servers that were designed from the scratch to be easily scalable by just
adding servers. Which is the case for most scripting languages PHP/Ruby/Python
et al and for well designed JVM/CLR/native/Go applications as well. However, I
would argue scaling the app boxes/containers is the EASY part. Again, you can
always scale up by "just" copying machines (I know, it's always harder that
that). The hard part comes with managing the database servers or your cluster
of messaging queues, or some other stateful thing that has to persist data
SAFELY. Does Kubernetes/Docker REALLY make my life easier with those kinds of
things? Is the answer to use DynamoDB/BigTable/RDS/managed queues and forget
about that hassle of managing a database or a queueing cluster? Looking for
answers :) . Thanks!

~~~
m_mueller
> I've been wondering how is that fundamentally different from launching new
> instances via AWS/Rackspace/Google API? Is there really a fundamental
> difference between shipping a Docker container to a set of servers, or just
> relaunching/rebuilding servers?

There's two perspectives to this:

* From a user's perspective, ignoring performance implications, containers should be the same as VMs.

* From a hardware perspective however, containers are much more lightweight in terms of CPU, Ram and Disk space, since they all share at least the kernel.

Why should you care? Well if something is less resource consuming, it means
that (a) you could run it for less than what you pay for VMs or (b) it can be
sold to you for less. There's some additional benefits like fast "boots",
strict decoupling of persistant and non-persistant storage (which I find an
advantage, restarting a container cleans up whatever you don't care for), but
at the end I think it comes down to money.

~~~
ecnahc515
To add to this. Sure you can spin up ec2 instances and you don't have to worry
about atomicity. However, what happens if you want to run more than one
service per VM? That's where containers allow to take full advantage of the
system.

------
wyc
> we set out to build and deliver Google’s infrastructure to everyone else

This statement rings pretty true as Kubernetes (also known as k8s) has some
Google biases. Not all cloud providers will have such an easy time providing
all the infrastructure necessary to run CoreOS + k8s smoothly.

For example, Kubernetes assigns each Pod (k8s unit of computation) an IP
address, which is only simple to do if your cloud provider supplies something
like a /24 private block to your nodes. CoreOS came up with the VXLAN-based
Flannel project to make this model more portable[0], but Layer 2 over Layer 3
isn't something I'd like to throw haphazardly into my production environments.
Google Compute Engine conveniently provides this setup as an option.

Another example of Google-favoritism is the strong preference of centralized
storage--particularly GCEPersistentDisk. At first I was concerned about
centralized storage by default as we know disk locality is a Good Thing (TM),
but after reading a paper that claimed networking is improving faster than
disks are[2], I felt somewhat better about this. However, it's still pretty
obvious that a Google Persistent Disk is the way to go with k8s[3].

That said, I'm really happy that Google has open-sourced this project because
it is indeed a functioning, tested, and easy-to-use distributed system. I'm
sure that the devs aren't aggressively shutting out other cloud providers and
that these biases are probably just a side-effect of their resource allocation
process and the problems that they intend to solve (e.g. GCEPersistentDisk
used to be a core type instead of a module--it has since gotten better). It's
still important to evaluate a technology's biases and potential evolution
before throwing your product on it.

[0] [https://github.com/coreos/flannel](https://github.com/coreos/flannel)

[1]
[https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/volumes.md)

[2] [http://www.eecs.berkeley.edu/~ganesha/disk-
irrelevant_hotos2...](http://www.eecs.berkeley.edu/~ganesha/disk-
irrelevant_hotos2011.pdf)

[3] Do you see any other providers here?
[https://github.com/GoogleCloudPlatform/kubernetes/tree/maste...](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/pkg/volume)

~~~
justinsb
I'm working on adding AWS support for Kubernetes. I just last week finished
Load-Balancer (ELB) & Persistent Storage (EBS) support, and they're currently
going through the pull-request review process. Once they merge (I'd guess a
week or two?), AWS will be on-par with Google Cloud Engine feature-wise.

I have found the Kubernetes team to be nothing other than extremely supportive
of efforts to support AWS & non-Google platforms. It takes a little longer to
translate some of the Google-isms to other platforms, but I'm happy for the
thinking behind those decisions, vs just adopting lowest-common denominator.

~~~
josephjacks
I also see quite a lot of IaaS providers here:

[https://github.com/GoogleCloudPlatform/kubernetes/tree/maste...](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs/getting-
started-guides)

------
tdicola
Congrats to the team! Interesting to see a big focus on Kubernetes going
forward--doesn't CoreOS kind of compete with Kubernetes? (even though
Kubernetes builds on CoreOS tools like etcd)

~~~
fredkelly
Also interested how the Rocket/Docker debate will fit in with this?

~~~
kelseyhightower
This is a great question.

CoreOS is currently working on adding support for Rocket[0] to Kubernetes.
There is still a lot of work to do here, but things are progressing nicely.
Kubernetes is a true opensource project and accepts contributions that makes
sense. Supporting another container runtime engine makes sense in terms of
choice for the community and forces clean separation between Kubernetes, the
cluster manager, and the underlying container runtime engine.

When it comes to Rocket vs Docker I think the competition will be good for
both projects and the community as a whole. At the end of the day, both
projects and the communities behind them want people to be successful with
containers.

[0]
[https://github.com/GoogleCloudPlatform/kubernetes/pull/5518](https://github.com/GoogleCloudPlatform/kubernetes/pull/5518)

~~~
Alupis
> both projects and the communities behind them want people to be successful
> with containers.

I think it's better said, Docker wants people to be successful with
containers, so long as it's Docker containers.[1][2][3]

We've seen a lot of very open work coming from the CoreOS team with Rocket and
the ACI... unfortunately we've only seen negativity and mud slinging coming
from the other camp.[2][4][5][6][7]

Docker is more interested in having an "open" implementation, rather than an
open specification. The things is... an Open Specification is far more
important for industry changing technology like containers.

[1]
[https://news.ycombinator.com/item?id=8683705](https://news.ycombinator.com/item?id=8683705)

[2]
[https://news.ycombinator.com/item?id=8938409](https://news.ycombinator.com/item?id=8938409)

[3]
[https://news.ycombinator.com/item?id=8789181](https://news.ycombinator.com/item?id=8789181)

[4]
[https://news.ycombinator.com/item?id=8938066](https://news.ycombinator.com/item?id=8938066)

[5]
[https://news.ycombinator.com/item?id=8938176](https://news.ycombinator.com/item?id=8938176)

[6]
[https://github.com/docker/docker/issues/10643](https://github.com/docker/docker/issues/10643)

[7]
[https://github.com/docker/docker/issues/9538](https://github.com/docker/docker/issues/9538)

~~~
curun1r
Don't kid yourself...it's been negative on both sides. CoreOS called Docker
fundamentally flawed, though they've backed away from that a bit.

From what I can tell, CoreOS was happy to let Docker, Inc control the
container platform and build orchestration tools on top of Docker, since it's
pretty clear that no one is willing to pay for the base containerization layer
and the only money to be made is in orchestration. But then Docker released
their own orchestration strategy and that started the current period we're in
where both companies have come off as being very unprofessional and concerned
only with their own future success.

Between Docker trying to build almost everything into the actual Docker
executable (it's a container runner, API server, build tool, process monitor,
cloud provisioner and orchestration tool...those last two are only not the
case because they got called on the strategy) and CoreOS's pointless pushing
of rkt (rkt itself isn't pointless, but the PR to add rkt support to Docker
was, as was implementing Docker support in rkt). Both sides have foolishly put
their own interests ahead of the container ecosystem.

To say that either of the two companies has been the bad actor in all the
drama would be biased. They've both come across like spoiled little children
trying to get their way. As someone who has been pushing my company (~8000
employees) to consider implementing a containerized deployment strategy, it's
been hugely inconvenient to have to concede to people arguing against
containerization all the points around how unprofessional the companies in the
space have been, both in behavior and coding practices.

~~~
fredkelly
+1, This has been my interpretation too, I feel like a lot of the rkt PR was
subtly aimed at Docker, it was very much "this is why it's better" rather than
"this is why it's good".

~~~
politician
Honestly, as a consumer of both projects, I'm happy that rkt exists because it
provides a working demonstration of a solution where container images are
orthogonal to other concepts such as orchestration. If it went the other way,
our vendors would be swearing up and down that the only way for this container
thing to work is if we install their complete solution.

I love my Apple MacBook Pro and its soldered components, but I need my data
center to be meaningfully upgradable.

~~~
ecnahc515
I love Docker, but it feels a lot like the days of incompatible VM images,
where you got super locked in, and if you tried anything that wasn't what they
had designed for, then you got screwed.

------
kungfooguru
Aren't they doing the same thing people have been complaining about Docker
doing? Running with features but not stablizing? Is CoreOS not as unstable as
the impression I've gotten of it from comments led me to believe?

~~~
kelseyhightower
It's true, the CoreOS team has been moving really fast. It's what startups
have to do in the beginning -- before the money runs out :). While our
announcement today introduces Tectonic and some funding, it also communicates
that we have found our market fit. This means we have the right set of
features to go to market with, and add value by making people successful with
containers.

Now it's all about execution.

CoreOS is now focused on balancing new features with maturity and stability.
For many of our projects this focus started almost a year ago. Take etcd for
example. CoreOS spent the last 9 months effectively rewriting etcd and it's
raft implementation to meet the stability requirements we need going forward.
Other projects have demonstrated great signs of stability such as CoreOS
Linux, our Linux OS focused on containers. It's not perfect, but there are
many people that run CoreOS Linux in production today. We expect this trend to
continue.

I should also note that we have many more team members employed by CoreOS than
we did a year ago. It takes time to build the right team, but so far I think
we have done a fantastic job. I'm still amazed at the number of projects we
actively maintain and ship with a team our size. Now we have the people power
to make our projects/products solid.

Thanks for raising the question and hopefully we can continue to answer it
with actions and shipping products that you will fall in love with -- or at
least get the job done without the pager going off.

~~~
cbsmith
> CoreOS spent the last 9 months effectively rewriting etcd and it's raft
> implementation to meet the stability requirements we need going forward.

Yeah, I noticed that. It almost seems like it would have been better to just
glom on to Zookeeper or some other directory service that was a bit more
established, just for the sake of being able to work on other things for the
last 9 months. Do you ever look back in retrospect on that?

~~~
jpgvm
The reality is that etcd is still probably a year or 2 away from being
production ready.

In the mean time Zookeeper has shipped dynamic ensemble configuration which
bring parity to the only thing etcd had any advantage in. (Not that it
-really- mattered, most people that were running dynamic ensembles were
already using Netflix's Exhibitor)

Zookeeper is also faster, has more features and you need it anyway if you are
running SolrCloud, Mesos, Hadoop etc.

~~~
cbsmith
Zookeeper definitely has some advantages both in terms of maturity and
features.

I'm not sure though that Zookeeper is universally faster than etcd (certainly
as you add nodes it slows down... though most people don't need more than 3 or
5 nodes).

Dynamic ensemble configuration is definitely a place where Zookeeper was
lacking before, but it isn't the only difference that matters. The big
difference I remember with etcd was partition tolerance. Last I checked
Zookeeper was all on the consistency side and it's partition tolerance
basically just wrote off the smaller partition. Either choice can be a good
thing or a bad thing for a directory/orchestration store, but particularly as
you scale up to large numbers, partition tolerance semantics seem to better
match the use cases.

So I don't think etcd is all inferior to Zookeeper, and I understand the
desire to build a system where all components are fundamentally AP, but
sometimes having a bit of impurity and some duct tape hackery in your model
can help you get out the door faster and focus on more significant technical
challenges that will help you build your community. Once you have a community,
you have more resources to address that duct tape.

------
chuhnk
Kubernetes will become a dominant container orchestration platform and API.
It's what Mesos should be, but with the backing of Google, kubernetes is more
likely to push into that space a lot quicker and with a lot more momentum.
Notice, there are a lot more startups forming around kubernetes than mesos.
Even still, competition is healthy, having multiple forms of large scale
orchestration and resource management systems are important for the tech
ecosystem. I imagine we'll see a couple more over the next few years. Docker
with their own. I'm sure we'll see one to solve these problems at a smaller
scale that is easier to manage. And perhaps another focused on a niche area
like AWS lambda functions but as an open source project.

~~~
abhay_agarwal
Kubernetes is not really a container orchestration platform. It helps organize
containers into logical groups with dependencies, but orchestrating resources
on your physical machines is well within Mesos' domain. Keep in mind that
Kubernetes and Mesos aren't mutually exclusive. Kubernetes has been
[implemented]([https://github.com/mesosphere/kubernetes-
mesos](https://github.com/mesosphere/kubernetes-mesos)) on Mesos.

My opinion is that Kubernetes is closer in identity to Docker: tons of OSS
excitement, many companies jumping on board, and really pluggable. As to
whether its going to become the critical infrastructure piece that gets
enterprise to pay $$$... that's a stretch.

~~~
josephjacks
Abhay, perhaps you can disclose in your biased answer that you are employed by
Mesosphere?

~~~
notacoward
Perhaps you should have disclosed in _your_ answer that you're a founder of a
Kubernetes-focused startup.

~~~
josephjacks
Noted! Apologies.

~~~
notacoward
Thanks, and cheers. :)

~~~
josephjacks
Cheers. :-)

------
achanda358
There seems to be some feature overlap between fleet and kubernetes. Anyone
knows are they planning to reconcile those two?

~~~
jsprogrammer
What more do you want fleet to do? It's a simple scheduler and seems to be
pretty much complete.

Expect CoreOS to shift resources towards proprietary revenue streams while
allowing the community to pick up slack on open projects.

~~~
kelseyhightower
I hope not!

I can tell you first hand the majority of our team members work on our
opensource projects. It's one of the main reasons I came to work for CoreOS
over a year ago. But the concern is still valid -- will CoreOS really keep
funding our opensource projects or will CoreOS pull a fast one and go open
core?

So far our actions prove we are committed to opensource and that continues
with todays announcement. We wanted it to be really clear that CoreOS is all
about opensource projects and collaboration, even if that means our
competitors can compete with us, that we took major effort in keeping the two
brands separate. There is coreos.com for "Open Source Projects for Linux
Containers", and tectonic.com that combines those projects in a commercial
offering. There are some non-open bits in the commercial offering, but they
don't conflict with the opensource projects.

You are right that we will allocate more resources towards making money, which
is required if we want to continue the good work we are doing on the
opensource front. You are also right in that the community will pick up the
slack, but please keep in mind that CoreOS is part of that community and we
plan to continue leading the way.

~~~
politician
Does k8s replace fleet in a tectonic system? It's been asked several times on
this thread, but there hasn't been a straight-forward answer yet.

~~~
jsprogrammer
I don't think they want to say anything that would potentially alienate
anyone, which is why you're unlikely to get a straight answer.

fleet will exist indefinitely, but kubernetes essentially replaces it with
something (promised to be) better.

------
donflamenco
How does this compare to Redhat's Atomic Host?

~~~
kelseyhightower
It's better to compare RedHat's Atomic Host with CoreOS Linux. Both projects
are focused on providing a slimmed down Linux OS focused on running containers
in a server environment.

Tectonic would be closer to RedHat's Openshift project/product, but that's not
quite right either since Tectonic is not a full blown PaaS solution.

~~~
jacques_chester
> _Tectonic is not a full blown PaaS solution._

Granted, but let's be serious for a moment.

Nobody with widely-recognised components is going to just _stop_ moving up the
value chain to selling turnkey systems. That's where the serious bucks are
waiting.

I expect that before too long, my employers' sales team will be seeing CoreOS
and Docker sales folk as well as the existing competition from Apprenda,
Salesforce/Heroku, IBM and Red Hat.

(And, again, I don't speak for my employers).

