> spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script
I never realised I was using spooky arcane oldhat stuff! I feel wizardly now.
My projects (for small clients and myself) basically use this.
- A "build.sh" script that does a local build of back end and front end
- A "deploy.sh" script that scp's everything to the server (either a digital ocean VPS or an EC2 instance), runs npm install, runs database migrations and restarts with pm2
So running my entire CI pipeline in terminal is: ./build.sh && ./deploy.sh
If you like this style of deployment, and you haven't looked into Ansible yet, give it a try, you may enjoy it. I had similar build processes before using Ansible and I have found it a really positive step up.
My favorite deploy is using a simple `git pull` instead of scp. I also avoid complex build tools when I can. You either need your built files in the repo or you need to avoid them. Either are fine for my small personal projects. The only real exception I make is keeping any necessary secrets out of my repo. Those get dropped on the server manually.
This also solves the roll-back problem mentioned elsewhere. Just checkout the previous version (as long as you don't make any side effects like DB updates that are incompatible).
I admit, this requires relatively simple software.
Naive question: what's the correct way of doing this when you need to cp your content to a dir like /var/www/html that your user doesn't own (when logging in as root is prohibited)? My "spooky" (and probably very stupid) method is an Expect script that lets me supply a password to sudo.
Ideally you want some atomic deployment strategy, rather than having to deal with whatever crap might have been in the directory previously. Something like - extract your deployment artefact to a new, unique directory - and then read-only bind-mount that directory as /var/www/html.
> and probably very stupid) method is an Expect script that lets me supply a password to sudo.
The old school sysadmin way of doing this would be to have a dedicated deployment user (ssh pubkey auth only - possibly restricting only specific commands), with a sudoers[.d] configuration to allow that user to run an explicit group commands without a password (NOPASSWD)
Usually these directories are owned by another group (www or www-data) so that your web server program (nginx, apache, etc) can access them without running as root. If you add your user to that group, you should be able to manage files in your web root without sudo. Be careful with permissions, though - you may need to chown to set the group after copying so that the files are readable by your web server.
a pattern I have seen before is to clone the branch commit of each release to a folder /var/releases/{release_commit} and then html is not a folder but a soft link so /var/www/html -> /var/releases/latest_release_commit
this is useful if you need to revert back quickly.
But then releases folder needs to be cleaned up, or you can run out of space.
Safer/easier to keep the .git somewhere else entirely, rather than in the working directory. Git's got arguments that let you do that. Then you can check out to a dir that doesn't have the .git dir in it at all.
Alternatively, it's pretty easy to use git-archive (optionally, with an exclusions list to keep out files you don't want to serve publicly) to do something similar without ever cloning the remote repo. You can fetch a zip or tar.gz with a given commit's files, straight from a remote repo.
This is a great setup if you work with a lot of clients. Every client we work with wants everything to run on their infrastructure that they provide, and then they provide the weirdest jankiest setup you could imagine. Because of this I keep everything basic if it cant run on a fresh linux install or less (looking at you docker) then its going to cause issues down the line.
Basically same here. In cases the deployment has to be zipped and manually deployed to elastic beanstalk. On some EC2 instances for projects running a PHP stack, I don't even bother with this, just SFTP script changes to the server straight out of my IDE. Take care of any rollback needs with local Git versioning.
I had a similar setup a couple of years ago, where I had to deploy without downtime. My solution was to simply have the old version running in parallel until I was certain the new version was ok
----------
Static website:
1. Setup NginX using a symlink to the current version
2. Copy the new files to a separate folder
3. Change the symlink to point to the new version
----------
REST service (behind NgninX reverse proxy):
1. Current version runs on port X.
2. Deploy new version and run it on port Y.
3. Update NginX config to use port Y and reload
4. If you need rollback, just reverse step 3.
This can be done using scripts or Ansible too if necessary.
Keep a simple tool like Ansible around for those spooky admin tricks and you can take advantage of the Ansistrano plugins for smooth deploy and rollback (Ruby's great Capistrano tool ported to Ansible).
I've been writing some Ansible playbooks recently for the first time in years, and came upon geerlingguy's work. That guy is a powerhouse when it comes to writing Ansible roles/modules!
That's sort of pet peeve of mine: Migration are done separate from code deploys. Version 1 of your code runs on schema version 1. Schema version 2 does not make chances that will break code version 1. Code version 2 can utilize the chances made in schema version 2, but you're still able to rollback the code.
Each schema migration should also come with its own rollback script.
The downside is that you might need three migrations for some operation, but at least you won't break stuff.
The assumption that you can do a schema migration while deploying new code is only valid when you have very limited database sizes. I've seen Flyway migrations break so many times, because developers assumed it was fine to just do complicated migrations on a 200GB database. Or a Django database migration just lock up everything for hours because no one cared to think about the difference between migrating 100MB and 100GB. And I've never seen anyone seriously considering rolling back a Flyway migration.
Agree with this and have practiced and advocated for it. Make the schema changes to support the new feature first, then verify existing software still works. Deploy the schema change. Then develop the new feature, test, and deploy the program. That way you can deploy and rollback without needing to synchronously run a bunch of migration routines.
I never realised I was using spooky arcane oldhat stuff! I feel wizardly now.
My projects (for small clients and myself) basically use this.
- A "build.sh" script that does a local build of back end and front end
- A "deploy.sh" script that scp's everything to the server (either a digital ocean VPS or an EC2 instance), runs npm install, runs database migrations and restarts with pm2
So running my entire CI pipeline in terminal is: ./build.sh && ./deploy.sh