This article fails to mention the consequences of the rush to push version B, rather than rollback to resolve. As far as I can tell the result met the needs of the business?
If it did that's pure luck. It could have also had another bug. Rolling back keeps your service up. Pushing forward faster maybe gets your service back up, or maybe breaks it some other way.
When dealing with a complex system, roll back first, then figure out why it broke and how to fix it for the next attempt.
I should clarify - I don't disagree that rolling back and fixing the bug (or in a continuous delivery world - fixing forward as a priority) is the right thing to do. I'm just saying the article leaves out a lot of details and fails to articulate the reasons why the road taken was bad.
> As far as I can tell the result met the needs of the business?
They have burnt their bridges by making it impossible to revert to older version. In effect, if someone can not run a newer version, they will be forced to completely abandon the software.
Of course, that still counts as "meeting needs of the business" if the only thing your business needs is surviving for a couple more days. But that can be accomplished a lot more easily by doing nothing — your business is unlikely to vanish in puff of smoke tomorrow, as long as you simply sit on your hands.
IMO, they could have halted distribution of buggy newer version, released minor fix for older one to stop it from crashing, then finally a backwards-compatible major release. That's more or less same effort, but with less damage to reputation and users of software.
I thought the same thing. It's obviously a bad situation, but getting A out of service and getting every instance on version B seems like pulling victory out of the jaws of defeat.
My guess is there is some detail missing that clears up why what happened was wrong. Sounds like A had a bug that rose to the level of being a security problem of the denial of service variety, so redeploying it seems like a terrible idea.
