
Don't deploy on Friday and 3 other unwritten rules of software engineering - ben-clubhouse
https://clubhouse.io/blog/dont-deploy-on-frida-3-other-unwritten-rules-of-software-engineering
======
Sarki
Point "2\. Maintain regular backups." should rather be "2\. Maintain regular
backups and validate your restoration process."

How painful it can be when you realize that your backup process was wrong from
the beginning and you're left with a useless gigabytes sized blob when in need
to restore your DB...

------
cphoover
hmm I dunno... If you have CD working correctly you should be shipping
multiple times a day depending on the product. If there is a problem with the
deploy it should be as simple as rolling back to the container image of the
last version...with K8s or ECS this takes seconds.

I really hate the whole "let's plan for a midnight deploy" paradigm... and
find it very unconstructive.

~~~
sbergot
How do you rollback a DB schema change?

~~~
clintonb
You deploy in phases. You only remove data in the final phase. I wrote about
this at [https://engineering.edx.org/django-migration-
donts-f4588fd11...](https://engineering.edx.org/django-migration-
donts-f4588fd11b64).

------
koneles
It's funny because I'm reading this from my work, instead of doing somethink
productive

~~~
clintonb
If reading this increases/reaffirms your knowledge of good practices, that
_is_ productive.

------
ogn3rd
It's called Read-Only Friday for a reason. Unless of course you enjoy spending
your weekends debugging and rolling back. No thanks.

------
aliswe
"Slack is famous for its role in increasing connectivity between team members,
but, at what cost (aside from the monthly fee)?"

And what's bikeshedding?

~~~
g-erson
> The term was coined as a metaphor to illuminate Parkinson’s Law of
> Triviality. Parkinson observed that a committee whose job is to approve
> plans for a nuclear power plant may spend the majority of its time on
> relatively unimportant but easy-to-grasp issues, such as what materials to
> use for the staff bikeshed.

\--
[https://en.wiktionary.org/wiki/bikeshedding](https://en.wiktionary.org/wiki/bikeshedding)

------
curtis
A corollary to "Don't deploy on Friday": "Don't deploy on Monday."

Unless you really want to work over the weekend.

~~~
lojack
Why would deploying on a Monday result in working over the weekend?

~~~
curtis
I guess it depends on why you're doing the deploy. If the whole point of doing
the deploy is to pick up new code changes, then the people responsible for
doing those code changes run the risk of having to work over the weekend to
make sure they're ready. This can be a big problem at (for example) startups
where the same engineers are responsible for writing new code and doing
deploys. But it can happen at organizations of all sizes.

If all you're doing is rolling out a new version of Postgres and it's supposed
to just work, then a Monday deploy makes perfect sense. However, as a software
engineer, the primary reason I've been involved in deployments is because
we've written new code and we need to release it. In an ideal world, the code
would be completed _and_ rigorously tested by no later than Friday at 5pm. In
the not-so-ideal world we actually live in, this isn't always true, which is
why I prefer to target deploys for Wednesdays.

~~~
sbergot
If you were working on a weekend on a feature and I were the release manager,
I would not be confident in your delivery and I would ask you to wait for the
next release in order to test calmly your changes.

~~~
tracker1
You assume management is generally "sane." There's a reason why the likes of
Dilbert's Pointy haired Bos (PHB) often ring true.

