
Kubernetes at Box: Microservices at Maximum Velocity - robszumski
https://www.box.com/blog/kubernetes-box-microservices-maximum-velocity/
======
avitzurel
I haven't used kube in production yet.

However, I'm using mesos, marathon and chronos to manage a production
environment with service discovery glue based on Route53.

Using Docker to ship an application to a well configured environment is just a
delight, the amount of configuration needed is absolutely minimal.

However, I think people need to realize that it's "easy" if your services are
not talking to each other and dependent on one another in a way. If service X
is using service Y directly (via HTTP), it gets a bit more challenging.

The way I like to configure micro-service is based on messaging so you send a
message to a queue and multiple satellite services can consume that message
and do stuff with it.

If your services are dependent on one another, the configuration gets trickier
and the maintenance gets a bit harder.

Good job by Box also contributing back to the core of Kube based on what they
needed, based on it getting merged I am guessing other people will find it
useful as well.

~~~
TheIronYuppie
Thanks for the feedback - may I ask if you've looked at Kubernetes at all? It
appears to do exactly what you describe, and I'd love to hear if you evaluated
it but found something missing. You can see some details of how this would
work in a recent blog post by CoreOS
([https://coreos.com/kubernetes/docs/latest/services.html](https://coreos.com/kubernetes/docs/latest/services.html)).

Disclosure: I work at Google on Kubernetes.

~~~
avitzurel
I have looked into Kubernetes and am running a couple of POCs on it.

I agree 100% Kube answers everything I am missing with mesos/marathon
combination, that's why I am planning to start moving over new services.

------
tedreed
Anyone working on K8s at Box or I guess anywhere else that has deployed it
partially feel free to answer this, but:

How do you handle gatewaying traffic into Kubernetes from non-K8s services?
I've been trying to get a basic cluster out the door with one of our most
stateless services, but I'm having a having a hard time just getting the
traffic into it.

The mechanism I'm using is having a dedicated K8s nodes that don't run pods
hold onto a floating IP to act as gateway routers into k8s. They run kube-
proxy and flannel so they can get to the rest of things, but ksoftirqd
processes are maxing CPU cores on relatively recent CPUs trying to handle
about 2Gbps of traffic (2Mpps) which is a bit below the traffic level the
non-k8s version of the service is handling. netfilter runs in softirq context,
so I figure that's where the problem is.

Are you using Calico+BGP to get routes out to the other hosts? What about
kube-proxy?

~~~
sporkland
I work at Box on this project.

Our network setup is constantly evolving due to a number of internal
networking limitations related to nearly static ip-addressing and network
acls. I'll describe our current setup and then describe where we'd like to go.
The big piece of context is that we already have a number of services already
being managed via puppet and a smaller number of new and transitioned services
in Kubernetes so we need to allow interop though a number of different
mechanisms.

We are currently using Flannel for ip-per-pod addressability within our
cluster. No services are communicating inside the cluster so they aren't using
kube-proxy yet. For services outside the cluster talking into the cluster we
are using a heavily modified
([https://github.com/kubernetes/contrib/tree/master/service-
lo...](https://github.com/kubernetes/contrib/tree/master/service-
loadbalancer)) which we have contributed back yet. It supports SNI and virtual
hosts. And we get HA and throughput for the individual loadbalancers by using
anycast.

We have a number of internal services outside the cluster slowly moving to
SmartStack. So I assume we will be figuring out interop with that and running
it as a sidecar at some point. We would like to move to calico as we have some
fairly high throughput services running outside of the cluster which we need
to avoid bottlenecking on a loadbalancer for. We have separate project running
internally to move our network acls from network routers to every host via
Calico.

Hope that is more helpful than confusing.

~~~
tedreed
Thank you for that answer, it's helpful. We've also been considering Calico
but it seems like a fair bit of work and the project's pretty overdue as it
is.

------
geggam
This is the 1st use case I have seen where microservices are starting to make
sense.

My question is what about network security ? How is that part managed ?

~~~
sporkland
We are using a product called calico which integrates with kube, openstack,
baremetal to setup iptables rules which simulate network ACLs on all of the
receiving hosts so that services are only able to reach network endpoints
which are whitelisted for that service.

Disclaimer I work for Box on Kube

~~~
geggam
conntrack module cause performance hits or you disable some ? any tcp tuning
kernel bare metal side ?

------
TheIronYuppie
Really cool story - it's been awesome to see how Box has contributed back to
the community as well!

Disclosure: I work at Google on Kubernetes.

------
scient
Totally lost it at "e knew we'd ultimately need dozens (even hundreds) of
microservices to be successful" and did not read any further. I am having a
very hard time seeing that as a criteria for success, not to even mention
imagining how that mess is managed. Is this really common to have so many
microservices?

~~~
avitzurel
You can easily end up with dozens of services if you split up your application
aggressively enough. Think about all the parts in your application that are
handled by workers like Sidekiq/Celery, these can all be applications and not
part of the monolithic.

For example for us at Gogobot, every time a user submits a recommendation we
detect the language. This can be a service instead of a worker and the code
can live separately.

If you do this often enough and aggressively enough, you end up with dozens of
services. Once you have an environment that allows easily testing/launching
these it's much more efficient to launch a service than replicating your
monolithic and assign worker for things.

~~~
geggam
If your application / poorly architected then when you split it you just
spread that pain around to more systems

Design it then split if where it makes sense.

------
foota
I hope you all appreciate the fact that the kubernetes team was initially
required to order new hardware for spinning up services.

~~~
avitzurel
s/kubernetes/box no?

~~~
foota
s/foota/has no reading comprehension

