Networking in particular: with thousands of containers, now there are lots more interfaces, routes, conntrack entries, "background" traffic, iptables rules etc.
Some problems are algorithmic: for example historically a lot of code has been written with the assumption that the number of network interfaces is small. With >1k interfaces, suddenly O(n) lookups take time. Similarly, iptables rules are run sequentially. etc.
Some resources have limits. Exceeding these may impact performance. In his case the load blew up the ARP cache. Nice!
Am I missing something? 512gb / 2500 is around 200mb per container.
I figured that one hop over 100Mbit Ethernet to remote memory was going to be faster then swapping to spinning rust (remember this was before consumer SSDs, and onions on our belts), so I made a ramdisk on the old machine and mounted it over the network with the nbd (network block device) kernel driver, ran swapon on the nbd and boom, extra "512MB" of RAM.
It worked amazingly well, and (knock on wood) none of my roommates ever tripped over the Ethernet cables.
On a more serious note, gonna add that trick to my book - still plenty of rust spinnin' round
Case in point for me is Docker itself. By default it will write logs to json-file and not truncate or rotate these logs. Packages distributed for Ubuntu et al also don't set these limits, so unless you manually make it a system-wide default or set it for each container you run individually it will eventually eat your disk-space. This is a very dangerous setting since you're probably using Docker in a lot of cases where all you do is run ephemeral, stateless workloads. Having out-of-disk creep up on you on these kind of hosts is most unexpected. The same could be said of having the -p port forwarding option default to binding to 0.0.0.0 instead of 127.0.0.1. Also a footgun and an exposed ElasticSearch with PII waiting to happen...
We are talking about an ini file. This isn't rocket science.
However they will now be aware that said feature is insecure and should know the consequences of enabling it.
One of the things the MySQL developers did originally was to ship the code with three examples of the my.cnf config file for small, medium and large memory systems. I wish there was something like that for docker container density and OS tuning parameters.
I think what we need is basically a matrix of OS settings which will max out your docker density for N CPUs and M gigs of memory. There are also probably settings for network configs which depend on your link capacity also.
I can dream of a day when you can spin up an Ubuntu Docker-flavor server which optimizes itself to give the highest density of containers given the hardware (or VM) it's running on.
For the port-binding thing, I'd just remember that it binds to 0.0.0.0 when not explicitly specified otherwise, and then use docker network and not port-forwards unless absolutely needed. For example, if you have an application and a couple of backing services (database, redis, ElasticSearch), then only your application needs a port forwarded from the host, the rest can live within the docker network.
Maybe we could change the default based on api version.
Hopefully rootless docker completely eliminates the races by removing the kernel resource contention.
Wonder why Nginx didn't scale.
Consul-template writes a config and then does an action. In the case of nginx, I would assume the action is to send a SIGHUP. I think haproxy would have also been an option here, it has better support for srv record to do updates from and the like.
The only time our tried and tested consul-template + SIGHUP method is every unsuccessful (and we've ended up jusy having to put processes in place to stop this) is if we have the same nginx handling inbound connections to the cluster under high load and we try and respawn all the containers at once. Then things start to go wrong for 5 minutes or so then back to normal.
While "the occasions error response" isn't perfect, I suspect that for most use cases it's good enough, so I'd still be interested in knowing more specifically what happened to that nginx...
Time to start thinking about 128bit systems!
I'm a massive go and rust fan, but I don't expect the entire world to be re-written in them any time soon.
>> The density per server savings is probably a drop in
>> the hat compared to the cost of the engineers themselves
>> [memory] is the only thing I think you'd get from C++