
Ask HN: Who uses Kubernetes in production? - the_gigi
Kubernetes is making amazing progress. A lot of people are involved and even more people are watching closely. But, who actually walks the walk and runs production software in Kubernetes cluster?
======
leetbulb
I do! The hardware layer consists of physical machines running XenServer.
Networking layer is 1Gbit WAN interfaces and 10Gbit LAN interfaces on "virtual
switches" all wired together with pfSense. Gitlab-CI takes care of deploying
just about all of the fabric on top of that including the images and app /
system components / resources. Ingress is currently being overhauled, but
right now it's essentially exposed HAProxy (pfSense) on top of dedicated, HA
ingress VM's. Oh and it's all CoreOS. We're running three sites on it each
with dev environments. Maybe a couple random API's, too. I haven't looked
through all of the namespaces in a bit.

Edit: Gitlab-CI runners run on kubernetes as well using the dind images.
Ingress nodes will soon be given public IP's. Public IP's are currently on
CARP failover. After the gitlab-ci-multi-runner 1.1.1 release (allowing shared
artifacts) and Kubernetes Deployment resources (providing a way easier
deployment workflow and orchestration of pods), CI/CD is a breeze. We have
dedicated nodes for MySQL (PXC) and ZooKeeper because these don't play well in
the Kubernetes network environment - don't ask me to look at the examples ;)
Currently running with Flannel for the overlay, but we're evaluating Calico
and waiting on new Docker features to pull the trigger on something else...
Multicast, isolated namespaces, VLAN's would be awesome :)

Edit2: I don't know why I keep saying "we" ... I've built and run this thing
solo on top of programming... Not enough hours in a day...

~~~
TheIronYuppie
Can you drop me a note - Aronchick (at) google? We'd love to chat about your
deployment :)

~~~
leetbulb
You got it!

------
AkihiroSuda
Kubernetes in Production in The New York Times newsroom
[https://kubeconeurope2016.sched.org/event/67f2/kubernetes-
in...](https://kubeconeurope2016.sched.org/event/67f2/kubernetes-in-
production-in-the-new-york-times-newsroom)

------
chews
500 node clusters EVERY darn day. Glad they removed the 500 node the limit
since kube 1.2 to allow clusters of larger size. We run primarily on google
compute but we also run smaller clusters on Amazon.

------
techthumb
SoundCloud are moving to Kubernetes too:
[https://www.youtube.com/watch?v=5378N5iLb2Q](https://www.youtube.com/watch?v=5378N5iLb2Q)

------
hijinks
We are running it in production

AWS with kubernetes 1.2.1 and calico as the overlay network. We have all our
web apps in kubernetes and working on our background job apps next.

------
TheIronYuppie
We have tons of folks - any particular verticals I can highlight?

Disclosure: I work at Google on Kubernetes.

~~~
aprdm
Do you have examples of running it on premises without external internet
access?

------
nethier
We're running many small (10s of nodes) clusters on metal with coreos.
Networking is some in house stuff we've purpose built so we can get public ips
on the pod.

Internal adoption seems to be going well so hopefully this grows.

~~~
leetbulb
Check out the new Ingress stuff for exposing services to public.

[http://kubernetes.io/docs/user-
guide/ingress/](http://kubernetes.io/docs/user-guide/ingress/)

[https://github.com/kubernetes/contrib/tree/master/ingress/co...](https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx)

The ingress controller runs on nodes with a public IP. When you create an
ingress for one of your services, one of the controllers will pick that up and
expose it. The ingress controller linked above is built on nginx and
essentially just reconfigures nginx based on your ingress resource spec.
Similar to when you define a LoadBalancer in a service with GCE, the service's
external IP is set to the upstream ingress controller node's public IP.

------
cweagans
We use it at NBCUniversal for some internal things. Nothing public-facing,
though. Not yet, anyway.

------
yuvipanda
We run it at Wikimedia for our Tool Labs setup, and are migrating more things
to it over time.

------
trungonnews
Who's using google cloud container service?

