I don't "fully" automate. I partially automate, then build my workflows from these composite parts. Composition.
For a start, I file all my bash scripts/notes/snippets in Vimwiki. Zettelkasten is truly brilliant.
Then... I write almost everything I do INTO Vimwiki (git is for programs, Vimwiki is for snipppets. Vimwiki is managed with Chezmoi). Hence, whenever I need to do... anything... I can build that workflow out of past snippets from the Vimwiki filing system.
Over time, my workplace and home workflows are automated. You know it's automated once you find a "set" composition, you use for 2 weeks straight without variation.
I only bundle "set" workflows, which haven't changed, into discrete scripts.
That’s what the post’s approach gives you: a checklist you run in order, with some steps automated.
In hindsight, the biggest challenge that I've found is getting your "do-nothing script" to become the authoritative source of how to do something. The fact that the steps themselves are still manual means that it's easy for people to create their own manual side paths to cover new situations and it's hard to get those added back into the script.
If your CI/CD server is running all the steps then the only way to make something happen is to get it into the build script. If there are humans in the loop then there will always be new things that come up, people who do them in slightly different ways, and stuff that doesn't get committed back into the scripts.
Once the script actually automates a few of the more tedious steps, people are much more likely to use it. And when they use it, they're likely to help maintain it when new situations need it.
Eventually, you may end up with a script that does automate it all, and can be put in CI/CD. (Hey, I can dream, can't I?)
> And when they use it, they're likely to help maintain it when new situations need it.
Unfortunately, "likely" means "not always", which turns into something that is no longer a real source of truth, and you're back to people having their own undocumented way of doing things.
I've also found that it's easier to bikeshed when you're printing our commands for people to run. Either because they see the command more clearly or they have different dependencies installed.
It's still better than nothing, don't get me wrong. It's just become clear to me that if you don't build your "do-nothing script" with a real plan to get it automated (i.e. not just a dream that "we'll automate this one day") then it will quickly rot.
My only issue is that not automating the deploy is probably the worst thing to need to do manually. Having worked at a company that had a rather large and ever changing list of steps to complete in order to deploy (which took a 1-2 hours), it's something I find myself prioritizing now.
I personally like the name „checklist script“ (also checkscript) for that, which I found in that comments at the time. I think it conveys easily the two purposes of the thing (checklist gradually becoming a script).
But one side note: if you have sysadmin skills and can write some shell scripts and have used more than one distro, you are effectively a Do ker expert. Configuring a docker image is just says admin 101 with little bitty shell script fragments.
First, because automation takes some upfront effort to get done.
Second, because once it is set up it frequently acts to prevent change.
I find it is useful to consider alternative solutions to the problem like a checklist (which I call a program that runs on people) which allows you to set up a process in minutes (+ potentially a meeting with your team) and then get back to the project. Once the project has been running for some time you will be in a better position to automate it. First, you will already have some experience with running the process (with the checklist). And second you can plan the task of automating at lower priority and without it becoming a blocker in your more important project.
Yes, it is considered technical debt. Not all debt is bad.
Different teams might want to use it for different things, deploy different versions with it, etc. So it should be in a separate repo with its own tests and its own version string. Otherwise iterating on it is hell because you're doing so in a space that's ergonomic for app development, not deployer-tool development.
That way when your deploy target changes (goodbye helm, hello custom k8s operator), your users can keep using the stable deployer version while you work on the new one.
Following that, switching between the old way and the new way becomes an atomic action, which makes it much easier to triage issues of the "it worked with the old way but breaks with the new way" sort.
Or at least... I hope that's what I missed. Because if not I'll be reredoing it next year.
And it can actually be a mistake to try to jump from zero directly to fully automated nirvana.
If it's in the same repo, that means all of the features of the deployer can match the deployed code automatically - the deployer is made for and tested for that version.
With different repos, you add complexity. Now you have to support deploying multiple versions so there are more code paths and there's version checking etc. And more complexity also means you need more tests.
- you don't care about tests that span multiple versions of the app (e.g. patterns of usage around database upgrades)
- you don't care about git bisect sessions that evaluate a new test against old versions of the app
- you never want to try the new app on the old infra or the old app on the new infra
- you're not worried about tests written by people not on your team whose failures indicate scenarios that don't require action on your part (but might require action on theirs).
if it's in the same repo, you add complexity -- now I have to ensure my deployment of two unrelated code bases don't collide, there's more cross-team conflicts, and I have to clone down a 200gb repo.
The big problem I find is changing process in a big team is hard. One of the great things is make is usually installed on most developers machines. It reduces the friction when trying to get a team on board.
I am not suggesting you should give a toss, but at least admit what happens before you inflict the decision on your team.
Have a wonderful day =)
The critical observation is that these people won't always be around or have this deployment plan fresh in their minds. You need most of your deployments to follow the same kinds of rules, so that when you look at a misbehaving service that you haven't had to deal with in ages, you can figure out what's going on and why it's wedged. Every new thing it can do is tribal knowledge that's not captured anywhere. It might have already walked out the door in fact.
That's not the same as automating "everything", of course.
> It gets people into the habit of running a single command to initiate deployments
npx firebase-tools deploy --only functions -P staging
is also one command
> It’s a more obvious source of truth for putting the deployment instructions
If that command were put into CI/CD, that would be an even more authoritative source of truth.
> Because it’s already an executable script, it makes it easier for us to automate some of the steps listed
The executable script can't be run by CI/CD and is another potential source of bugs/mistakes.
The real "fake it until you automate it" for deployment would've been ssh'ing into the server, running git pull, and restarting the server. That's no longer possible with serverless nowadays, but this seems to go from taking something that's relatively straightforward to something that's very complex.
One common pattern that I'll see though is deploy.py not being top level so `python deploy.py` becomes `python scripts/deployments/staging/deploy.py` so you still get the benefit of not wondering where `deploy.py` is.
Another is the configuration file also having a default version that is not obvious. Something like `configuration/deployments/staging.yaml`.
Now you might argue that I'm exaggerating (and I think it's fair to say that some refactoring could be valuable), but the simple `python deploy.py` now actually is.
`python scripts/deployments/staging/deploy.py -c configuration/deployments/staging.yaml` which is significantly harder to remember than `make deploy-staging` for example.
However there is one non-trivial downside with (GNU) Make, and that is the non-visibility of env vars set with `export` when running `make -n`. That is, if your Makefile looks like this:
The output from `make -np` is incredibly verbose and isn't easy to filter with standard CLI tools, which makes this doubly frustrating. You basically need to write an AWK/Perl/Python program to parse it. If there was one feature I'd pay good money to add to a new version of GNU Make, it's an option to emit more-easily machine-readable output from `make -p`, or to ship a script in the GNU Make package that parses it into something standard like JSON or XML.
Make is great because its usually installed everywhere. The downside is that it has a lot of edge cases. Breaking out right away into a better language for the more complex tasks can make sense.
I'm really pretty OK with "if CI is down, and there's a prod emergency, and the test suite is broken", not everyone on the team can fix that. Escalate to someone who can, the 0.5 times this happens a year; the rest of the time, everyone clicks "merge" on their pull request and the code filters out into production quickly and efficiently. They don't want to think about it, but the customer gets their features as soon as possible. Not the worst thing. Way better than "our release person is on vacation, we'll restore your outage next week".
Even better, make this recipe's dependencies work so it doesn't do anything if you don't need to run init, then have everything else depend on init. That way no one needs to remember to run init first.
I like to also use the same commands in by build pipeline. It ensures I'm running the same commands as I do locally, with the added benefit of abstracting the deployment logic from what ever CI system you use.
Some issues I've found with this approach is having to install make on a build system, which these days is a slim docker container.
Some devs still run windows, but I think you can install some kind of equivalent.
Overengineering infrastructure early on is fun, but unecessary and something that can relatively simply be tackled later.
Most of your tools should run in CI/CD. That's the goal, and a pretty high priority one. But they can't only be run there.