
Running 1000 containers in Docker Swarm - titpetric
https://blog.codeship.com/running-1000-containers-in-docker-swarm/
======
barhun
2k nodes, 100k containers [https://blog.online.net/2016/07/29/docker-swarm-an-
analysis-...](https://blog.online.net/2016/07/29/docker-swarm-an-analysis-of-
a-very-large-scale-container-system/)

~~~
schmichael
5k nodes, 1 million containers, 5 minutes
[https://www.hashicorp.com/c1m/](https://www.hashicorp.com/c1m/)

(Disclaimer: I'm on the Nomad team but wasn't at the time of the post)

~~~
jacques_chester
I don't know much about Nomad and couldn't work out from the repo what the
jobs were. If I guess correctly, it's an app using Redis. Is that correct?

Disclosure: by coincidence of market forces, we're mortal enemies. Let's send
christmas cards!

~~~
schmichael
> it's an app using Redis. Is that correct?

Yup! Repo could definitely be clearer, but here's the code:

[https://github.com/hashicorp/c1m/blob/master/schedbench/test...](https://github.com/hashicorp/c1m/blob/master/schedbench/tests/nomad/classlogger/main.go)

Basically calls an increment in Redis and then blocks forever.

> Disclosure: by coincidence of market forces, we're mortal enemies. Let's
> send christmas cards!

Haha, hi mortal enemy! Christmas cards it is! If you're ever in Portland, OR
I'll buy a beverage of your choice as well. :)

~~~
jacques_chester
I extend the same offer in New York!

------
peterwwillis
I would use IPv6 for the orchestration network, probably not touch the tcp/ip
parameters except for port range (and open file descriptor), and break up the
broadcast domain into smaller networks. It is not advisable to have thousands
of machines on one broadcast domain, and it is a pain in the ass to
troubleshoot, not to mention causes bigger headaches when one network problem
affects all the nodes across the entire gigantic network.

------
eblanshey
Does anyone know how easy it is to set up autoscaling with Docker Swarm
running on Google Cloud or AWS? We're looking to get starting with Docker
Swarm or Kubernetes soon, and are considering using Docker Swarm because of
its simplicity and developer familiarity with Docker Compose (we use it for
our dev environment). We just want to add nodes to a cluster as traffic spikes
and subsides.

~~~
013a
I think most people would suggest that, if your use case is at a stage where
that is important to you, Swarm is not the right thing. Kubernetes or ECS are
better choices.

~~~
truetuna
I wouldn't recommend ECS. I've used it for a little over 6 months and even for
trivial things, it lacks. A couple examples that come to mind include not
being able to pass host environment variables into your container instances
easily, and not being able to specify that a service must run on all hosts.

Theres an open issue (made ~2 years ago) on GH for the 1st example and it
still hasn't been resolved.

------
hefeweizen
Slight nitpick, but this articles deals with "Docker Swarm mode" [1], which is
different from Docker Swarm [2].

[1]
[https://docs.docker.com/engine/swarm/](https://docs.docker.com/engine/swarm/)

[2] [https://github.com/docker/swarm](https://github.com/docker/swarm)

[3] Difference between Docker Swarm and Swarm mode:
[http://stackoverflow.com/questions/40039031/what-is-the-
diff...](http://stackoverflow.com/questions/40039031/what-is-the-difference-
between-docker-swarm-and-swarm-mode)

------
xchaotic
I always wonder, why not isolate on a process level, or even withing a single,
multi-threaded app. Sure you can run some sort of web service on hundreds of
docker containers or you can run a single, fast web server that scales?

~~~
acejam
When that single web server goes down, it's not so "fast" anymore.

~~~
patrickg_zill
Agreed, though I keep wanting to take the time to get VRRP working with a web
server to have redundancy. OpenBSD uses this to coordinate stateful firewalls
with 2 or more systems, if 1 goes down all state info is present on the second
node which takes over.

~~~
titpetric
Hi, OP author here: I have actually set up a VRRP (well, UCARP) on Docker, so
it's possible even to containerize this facet of running a HA ops stack with
Docker as the infrastructure. It is however, as you say, it is only used for
one active node + a number of fail-overs in case that one goes down. In terms
of maintenance (hosts do go down, scheduled downtime is common), it's
priceless to have this part of the puzzle portable as well. If you want to
check it out, there's a github available here:
[https://github.com/titpetric/ucarp-ha](https://github.com/titpetric/ucarp-ha)
\- and a future article with it is planned as well. It will also become a part
of the E-book which I'm currently working on and publishing on leanpub:
[https://leanpub.com/12fa-docker-golang](https://leanpub.com/12fa-docker-
golang) :)

~~~
patrickg_zill
OK so that runs on the host to the Docker instances. Pretty cool!

------
collyw
Can someone that needs to run workloads like this explain to me why this is
needed? It _sounds_ like over engineering for the sake of it. There are only
so many apps at Facebook scale in the world.

------
officelineback
I'm still interested in how to merge features like AWS Autoscaling with Docker
to right size the underlying infrastructure for the amount of container work
going on.

~~~
wwarren
[https://www.docker.com/aws](https://www.docker.com/aws) oughta be a good
place to start

