> whip through a thousand vms less than a second with this method
I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.
Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for.
One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.
Every company I've been with seems to re-invent the terms or swich their definitions slightly.
For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.
The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.
> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.
> synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.
Yup - I think that lines up.
> what is the master aggregate "switch" ? what does it do?
We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors.
In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on).
We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.
> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch".
I've never heard of this practice described as a canary before (shrug)
I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.