There’s an even faster strategy than this and it’s easier to setup.
You’re going to deploy 4 CI pipelines (so make sure you’re not manually putting together ci pipelines configs, use automation):
Pipeline 1: A conveyor belt of environments. All this pipeline does is spin up fresh environments then run a short automated smoke test. Hydrate the env with the most recent mask from prod. The trigger condition is there’s less than <Threshold> environments available. I did 8 on a whim and never saw a need to change it.
Pipeline 2: Normal garden variety CI pipeline triggered on merges to main. Output of this will be two artifacts persisted: a built package and your unit test evidence
Pipeline 3: Test your automated deployment by deploying the package build from #2 into the first of the queue of free envs from #1 trigger your end to end and integration and contract tests. Don’t run your security or operability tests here.
Pipeline 4: Async pipeline triggered on a 6hr schedule, do your long running stuff like fuzz testing here, your security tests etc. do these outside of the dev cycle.
Release candidates can only be signed after a successful run through 2, 3 & 4. That means prod deploys are on a predictable cadence which users and ops are usually appreciative of rather than we drop it in when it’s ready.
The DevEx is pretty sweet - you don’t see pipeline 1 or 4 in your build loop. Only the runtime of 3 would be comparable to the article - slightly faster than the article because no firecracker bringup overhead, no matter how small that is.
There are times when some corner of software development speaks a specialized language and this is an example.
1. Conveyor belt(?) of environments. Hydrate(?) the env(ironment). Mask(?) from prod(uction)
2. I think I got this. Typical "merge to main pipeline" with built product and test results as outputs.
DevEx(?). And not sure why I wouldn't see pipeline #4 in my build loop because I can't deploy unless 2, 3 and 4 pass.... Maybe you mean I don't wait to see it.
Also not sure how it's faster because environments still need to be brought up. Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds.
> Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds.
I read this as implying that there would be a "warm pool" of environments ready to go at any given time. This is an approach I've used in the past when environment construction was expensive (e.g. Windows servers), but there are certainly tradeoffs.
Not a technical term, i just meant a repeatable stream of environments being created
>> Hydrate
This one is a pretty common term, if you hydrate a schema then you just populate it with data
>> Mask
In many industries (health, finance, even more since GDPR and the ilk) production data can be sensitive, it might have Personally Identifiable Information for example. Masking is the process of taking a value from production and obscuring it but with plausible-looking data. It can be time consuming and gnarly.
>> DevEx
Developer Experience - usually i say DX but i stopped myself on that one when i was writing the comment!
>> not sure why I wouldn't see pipeline #4 in my build loop
It's part of your deployment loop for sure, but there's no reason it has to be part of your inner dev loop. You can continue on with your next task and be confident you'll start receiving telemetry from prod for your new feature's behaviour soon.
>> Unless you are trying to say that the environment is already running when the merge to master pipeline succeeds
Exactly this. The time for creating that env was spent async outside of your inner dev loop.
May I ask what stack you employ to meet these goals?
Many tend to reach for Gitlab CI or Github Actions but these piles of "executable yaml" never appear to be up to the task of complex deployment logic you describe in your post, not including that they don't account for multi-repo or composed artifact workflows naturally. The state of the art, if you can call it that, is Jenkins where you can drop into raw-ish groovy/java for the logic pieces when you need to. But then you run into the constant struggle of working around Jenkin's leaky abstractions and peculiarities.
You can patch together a pile of bash, python, go et al but you land in a worse place where there is no guiding structure to the automation for onboarding, enhancement, and maintenance.
I'm curious of other's experiences building complex build / deployment pipelines where up-front you have consistent entry structure to the automation but have all the escape hatches one would need to implement custom logic when required, in a type safe, potentially compiled, testable way (ie: pipelines as 'actual' code).
Of course one could write their own automation engine to avoid yaml hell and all that. However I am not seeing any pervasive solutions being presented that don't present "yet another (yaml | json | xml | cue | whatever) task dag launching containers running random scripts from wherever".
>> May I ask what stack you employ to meet these goals?
Nothing fancy :-) Ansible + Jenkins pipelines
>> these piles of "executable yaml" never appear to be up to the task of complex deployment
Agreed, there's an untapped market here - somewhere between your enterprise GitOps model like i'm alluding to (usually heavily integrated with Kubernetes) but repackaged for smaller scale use.
You’re going to deploy 4 CI pipelines (so make sure you’re not manually putting together ci pipelines configs, use automation):
Pipeline 1: A conveyor belt of environments. All this pipeline does is spin up fresh environments then run a short automated smoke test. Hydrate the env with the most recent mask from prod. The trigger condition is there’s less than <Threshold> environments available. I did 8 on a whim and never saw a need to change it.
Pipeline 2: Normal garden variety CI pipeline triggered on merges to main. Output of this will be two artifacts persisted: a built package and your unit test evidence
Pipeline 3: Test your automated deployment by deploying the package build from #2 into the first of the queue of free envs from #1 trigger your end to end and integration and contract tests. Don’t run your security or operability tests here.
Pipeline 4: Async pipeline triggered on a 6hr schedule, do your long running stuff like fuzz testing here, your security tests etc. do these outside of the dev cycle.
Release candidates can only be signed after a successful run through 2, 3 & 4. That means prod deploys are on a predictable cadence which users and ops are usually appreciative of rather than we drop it in when it’s ready.
The DevEx is pretty sweet - you don’t see pipeline 1 or 4 in your build loop. Only the runtime of 3 would be comparable to the article - slightly faster than the article because no firecracker bringup overhead, no matter how small that is.