
How to fuck up software releases - zdw
https://drewdevault.com/2019/10/12/how-to-fuck-up-releases.html
======
perlgeek
Another "fun" one: typo in git tag names.

The rakudo project has monthly releases with tags like 2019.07. Once I mixed
up the year and month in e release, so I created a tag 2018.06 instead of
2016.08 (don't remember the exact numbers), and pushed it.

I deleted the wrong tag as soon as I noticed, and pushed the correct one, but
of course when the date of the typo'ed release month came around some years
later, things blew up for everybody who still had the wrong tag in their local
repo (which turned out to be quite some developers).

~~~
m0xte
This is one thing I really loved about subversion. I just used the VCS
revision as the tag and version therefore avoiding this issue. Didn’t even
need to create tags.

That was until the version overflowed the 16 bit field it was stored in the
assembly manifest. Kaboom. :(

People spend way too much time arguing with semantic versioning when 99% of
the time the only important thing is the next number is bigger than the
previous number.

~~~
thrower123
I agree, it is such a cluster to get human-usable version numbers out of git.
I use a convoluted Rube Goldberg contraption based on tags and git-version[1].
It's so much less convenient than old SVN revision numbers.

[1]
[https://gitversion.readthedocs.io/en/latest/](https://gitversion.readthedocs.io/en/latest/)

~~~
e12e
My favorite is the complexity of having an in-tree changelog that lists
(shortlog) commits under appropriate tag headings - checked in under
appropriate tags.

It roughly amounts to:

1) commit fix (git commit -m "fixes #foo bug"

2) get hash/shortlog, add to changelog

3) commit changelog referencing commit in 1)

4) tag hash in 3) with new minor version 1.1.2 to 1.1.3

5) realize you forgot that 1) warranted a bump in minor version, edit
changelog to reflect new minor version

6) commit new changelog

7) move tag from 4) to 6) - hope you didn't push yet...

And people say RCS $Id:$ was a hack...

Now,the _real_ issue is actually to find a nice way to tag actual releases,
with current tag and hash. I realize this is mostly tricky when a project is
"checkout and run" \- without a build step (eg ruby on rails project). It's
still somewhat painful to make sure there's an up to date global variable that
correctly reflects the running version.

~~~
z3t4
I clone the project into a temp directory, then inject the version number into
the source.

------
vortico
I keep a release TODO list and about 50% of every release, I fuck up and need
to add something to the list. It's about 40 items long now (containing various
terminal commands and activities), which includes making sure manual entries,
documentation, the website, social media, dependencies, etc are handled
correctly.

~~~
PacifyFish
What kinds of things do you need to check for that aren’t automated? Maybe I’m
spoiled by our CI pipeline.

I’d love to take a look at your list

~~~
vortico
Here's a somewhat redacted version. `m` is my alias for `make`.

    
    
        - Finalize source
         - Legal check `LICENSE-dist.txt`
         - Update `CHANGELOG.md`
         - Bump version in `Makefile` and `Core.json`
         - Commit "Bump version".
        - Build (three OS's can be done in parallel)
         - `m clean`
         - `git pull`
         - Make sure you have the latest `Fundamental.zip` package in source root.
         - `m dist`
         - Manually test installer and fragile features (audio drivers, patch loading) for ~10 minutes.
         - `m notarize` (on Mac)
         - `m upload`
         - `git tag vX.Y.Z`
         - `git push --tags`
        - Release
         - Update version title and URLs in `Rack.pug`
          - `m upload`
          - At this point, normal users have access to new version.
         - Update server version in `config.coffee`
          - `m restart`
          - At this point, normal users will swarm to download new version. Keep an eye on server bandwidth.
        - Publicize
         - Twitter https://twitter.com/home
         - Facebook https://www.facebook.com/vcvrack/
          - Share on group
         - Forum https://community.vcvrack.com/c/announcements

~~~
enriquto
this list is a very simple algorithm. Why don't you tun it into a script
deploy.sh ?

~~~
tbrownaw
Some of the steps look like they'd be hard to do that for. Like for example
the one about making sure the license is correct, and the one about testing
parts that apparently don't have good automated tests / interact with the real
world.

~~~
enriquto
> Some of the steps look like they'd be hard to do that for.

Then those steps must be removed. I thought the mantra "deploy is ONE step"
was a more or less universally acknowledged truth.

~~~
creatornator
I mean unless a computer can check that all dependencies have compatible
licenses, that is unrealistic. It's a necessary step for publicly released
software to avoid legal issues

~~~
cyphar
Linux distributions (though I can only speak to openSUSE's process) have
automated scripts (which I believe are written by our lawyers) which check
whether the license of a package matches the license of the files inside the
package. It's how most package legal review gets done (and if the script can't
figure it out, it gets escalated to our actual lawyers too review. You cannot
submit a package to any one of the distributions we ship without the legal
review being approved.

So it is clearly possible to do -- and there are all sorts of tools which
figure out what SPDX license entries apply for every dependency (or vendored
dependency) of a given project.

------
jillesvangurp
A couple of practical solutions here.

1) Automate your releases and run that on a CI server if you can. "It works on
my laptop" is not ideal for releasing stuff.

2) Have a release branch, and only allow stuff to be merged that passes your
CI tests and trigger the automation under step 1 after a merge to your release
branch. This works for continuous deployment but can also work for releases of
other stuff. If you use semantic versioning, the CI server should tag the
release and publish the artifacts. That's one less thing that can go wrong. If
you forgot to bump the version number on your master branch, the build should
fail.

3) Have a release checklist. People forget stuff and having a small list of
"Are the release notes and readme updated? Is our CI not complaining? Have all
PRs been merged? Etc. Even with the above, premature releases through an early
merge to your release branch could happen. A checklist can prevent that. And
of course, if you can integrate those checklists into your CI. If your release
notes are unchanged since the last version

4) Release often. Small deltas are less work to test and less risky to impose
on users and you get feedback on stuff you did earlier.

------
lifeisstillgood
The big lessons from this article (which I love and resonate with)

\- You discover more bugs with your shins than your foresight

\- As long as you keep fixing it in code, you will eventually win - it's like
a video game, as long as you can respawn, you will always defeat the game.
Just keep buggering on.

------
josteink
> I am doomed to creatively outsmart my tools in releases.

That’s a very honest admission, and one I think lots of “why don’t you
automate _everything_ ”-crowd seems to fail to recognise as a real-world
factor.

~~~
praptak
I would say the article actually supports the "automate everything" position.
The author prevents future mistakes by adding more automation rather than
less.

~~~
rabidrat
and yet, future mistakes are still happening. A lot of work has gone into
automation but the outcome is often the same. So is it worth it?

~~~
josteink
Obviously the effort isn’t _worthless_ , but it’s not a silver bullet which
will magically “fix everything” either.

------
mcnichol
I love the humble nature of this post and how fallible we are.

I have spent many days playing whack-a-mole with issues and hacking out
solutions. What is equally great is the community sharing their gotchas and
how they solved them.

A wholesome and organic thread, thanks for taking the time to write it!

------
yitchelle
Don't do it on Friday.

On a serious note, I have found that the only way to not fuck up is to remove
as much human interaction from the release process as possible, which means
scripting. Even with scripting, if the inputs are bad, the output will also be
bad.

~~~
thrower123
Seriously, though, don't release on Friday. No matter how good your CI/CD
pipeline is, and how confident you are in your testing, you'll let some bad
releases through, and if you release on Friday, some poor bastard, whether you
or your coworker, is going to be left holding the bag, and have their weekend
ruined scrambling around to fix what is broken.

It's just inconsiderate. Things can wait until Monday.

~~~
Intermernet
I'm waiting for the next "disruptor" to make it a sales pitch: "Move fast,
break things, and only release on Friday!"

------
caseymarquis
Automating builds is fun! On the one hand I'd be worried if I joined a team or
contributed to a project which still had manual processes for release more
complex than 'run this script' or 'push to the x branch'. On the other hand,
that's a fun initial project.

When developers aren't (or are prevented from) improving their own
environment, that's a bad sign...

------
sroussey
Nice heads up! Working on a similar script to help with updating versions and
packaging across five platforms. At least I have a (mostly) monorepo. Which is
great until it’s not.

------
dbrgn
For bumping the version there's also bumpversion:
[https://github.com/peritus/bumpversion](https://github.com/peritus/bumpversion)

~~~
merlincorey
Two things should be noted:

1\. `bumpversion` is more general and has more functionality than `semver` in
the article 2\. `bump2version` is the currently maintained version[0]

I currently use `bumpversion` to manage versioning in applications with many
different versioning schemes as well as managing the versioning of their
deployment environments (such as updating Terraform files, etc).

[0]
[https://github.com/c4urself/bump2version](https://github.com/c4urself/bump2version)

------
mugsie
I really like how the OpenStack project does releases - it has a ton of
tooling, but for people who are deciding what to release, it is super simple.

e.g. - the latest RC of a project I maintain is L10-14 on
[https://opendev.org/openstack/releases/src/branch/master/del...](https://opendev.org/openstack/releases/src/branch/master/deliverables/train/designate.yaml)

by committing that file, the tooling chooses the right commit, does a tag, and
builds the python tarballs, and puts them on releases.openstack.org, and for
the client libraries, it pushes them to pypi.

it avoids a lot of the footguns of special scripts in repos, while allowing an
easy release process.

------
NKosmatos
I came here expecting to read about how Apple is (mis)treating their macOS
lately or how Microsoft has managed to fuck up their windows update process,
but instead I find a few good tips while building and numbering software.

------
RexM
This reads like the "Deploying on a Friday Afternoon Greatest Hits".

------
aepiepaey
> Update the signing script to save the tarball to disk (previously, it lived
> in a pipe) and upload these alongside the releases…

This should have been done from the start.

GitHub does not guarantee checksums for the generated source archives to be
stable, so they can change when GitHub updates their software (and yes, this
has happened).

------
tlocke
For Python projects I use the excellent Versioneer:

[https://github.com/warner/python-
versioneer](https://github.com/warner/python-versioneer)

It means you only specify the version as a git tag, and all the other things
that need a version number get it from that.

------
thedancollins
Phft, amateur. I've forgotten more ways to screw stuff up than this person
ever knew.

------
peterwwillis
A lot of these can be avoided by making all the steps repeatable. To do that,
you should run everything in a Docker container that's built solely from
version controlled files, and run it on a remote system that gets blown away
regularly. You can use Jenkinsfiles to specify exactly what to do with
different branches, and to really simplify things, you can have a different
job for every kind of release. Build in a dry-run option so you can run a
whole release without actually publishing anything. Each of your software
projects can have its own specific release steps in a script run by your
release container, while still following the same basic release process for
all your projects.

------
kissgyorgy
A thousand ways. One really good idea I read was that you write an empty
script, which prints the instructions for each step and you press enter, it
prints the next step. It can only print instructions at first, then later, you
can write code to automate some (maybe even validate things) steps. Something
like this:

    
    
        def git_tag():
            print("1. git tag the repository")
            input("Press enter to continue")
    
        def git_push():
            print("2. git push the repository")
            input("Press enter to continue")
    
        def pip_deploy():
            print("3. Deploy, run: python setup.py sdist upload")
            input("Press enter to continue")
    
        def main():
            git_tag()
            git_push()
            pip_deploy()
    

As simple as that! Then later, you could write code for every step as you have
time.

~~~
silvester23
I think this might be the article you read:

[https://blog.danslimmon.com/2019/07/15/do-nothing-
scripting-...](https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-
the-key-to-gradual-automation/)

~~~
kissgyorgy
Yes! Thanks! I couldn't find it.

------
kick
Can dang or sctb fix the title of this to the correct one?

"How to fuck up software releases"

------
greggyb
For some prior art in this space, it may be useful looking at ESR's shipper:

[http://www.catb.org/esr/shipper/](http://www.catb.org/esr/shipper/)

And some blog posts he's written on it:

[http://esr.ibiblio.org/?s=shipper&submit=Search](http://esr.ibiblio.org/?s=shipper&submit=Search)

