

Running MemSQL’s 107 Node Test Infrastructure on CoreOS [video] - carlsverre
http://blog.memsql.com/running-memsqls-107-node-test-infrastructure-on-coreos/

======
AdmiralAsshat
"At a recent CoreOS meetup, I gave a presentation on the design and
implementation of Psyduck, which is the name for MemSQL’s test
infrastructure."

So does the test infrastructure get exponentially more powerful when under
pressure, or what?

~~~
ankrgyl
Interestingly enough, in some ways it does. For example, during our nightly
stress runs the fact that the system is under so much pressure helps induce
race conditions that are otherwise pretty much impossible to reproduce.

------
kbar13
Awesome! I wish I had the opportunity to work with a large scale docker/coreos
deployment like this. I really perked up when you started talking about PXE
booting, I wanted to see what you guys were doing to manage updates.

I'm trying to make automated updates when using PXE[0].

When installed to disk, CoreOS's update manager automatically polls for
updates and downloads them in the background. On next reboot, the new version
is shifted into place. This is unfortunately unavailable when booting with
PXE.

~~~
carlsverre
Working on large server deployments is a blast! And CoreOS makes it even more
fun via its read-only focus.

In our system we don't do automated updates since we don't need that
functionality. In order to send out an update I download the image that I want
to deploy, patch it with CPIO (with my custom initialization code) and then
register it with DNSMasq (put it in a directory). At that point any machine
that reboots will start up with the new image rather than the old one. I
usually test one/two machines and then if they look good a script reboots the
rest.

If you wanted to do automated deployments, you could basically do the same
thing provided you have a server to manage the process. The master server
would simply poll CoreOS for new releases, and then do the same steps as above
to roll out the image. Incremental updates are trivial as well since you are
just rebooting the machines.

One thing to consider is running CoreOS out of memory rather than installing
it to disk. I find this is one of the best features of CoreOS since it makes
it even harder to think of a machine as any more than a temporary computing
resource.

Hope that helps! Good luck!

~~~
kbar13
My thought was to have CoreOS run the systemd cron job for update_engine even
when booting off of PXE, but to proxy[0] through corepxe (the jawn I'm trying
to write). This way I can MITM the request and make the request on behalf of
the host from corepxe, and take advantage of how omaha[1] works by returning
the appropriate status code and downloading in the background.

[0][https://coreos.com/docs/cluster-management/setup/update-
stra...](https://coreos.com/docs/cluster-management/setup/update-
strategies/#updating-behind-a-proxy)
[1][https://code.google.com/p/omaha/](https://code.google.com/p/omaha/)

~~~
carlsverre
That seems reasonable, however I am not super familiar with how update_engine
works. Likely your solution will work provided you setup your image to install
itself to disk and boot from there.

