
Friday Deploy Freezes Are Like Murdering Puppies – Charity.wtf - rbanffy
https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/?__twitter_impression=true
======
larzang
This post seems written assuming that engineers are calling all the shots. If
you work at a company like that, great, I envy you.

However in a company where engineering's desire for a stable deploy pipeline
with excellent visibility is matched up against business' insistence that all
dev hours go into features and SEO improvements and every sprint issue
regarding non-user-facing-improvements has to be fought for tooth and nail,
"don't deploy on Friday" is a very reasonable compromise for avoiding weekend
fires.

------
ubersoldat2k7
The question you should be asking is: what's so important that we need to rush
on a Friday? Can it wait to Monday?

Now, I agree with the premise of "trust your process" but if you've been into
this field for a while, you know things will break.

Once a QA guy I worked with, came back from his weekend with a new test
case/scenario that resulted in a critical issue right before we were ready to
deploy. Not even CI/CD can help you with that.

------
mharroun
My company has a rule of no deploys after 5pm (4pm on Friday) minus hot fixes
and other special cases.

We have unit tests, integration tests, CI/CD, launch new systems/features via
flagged code already in production, and we deploy 1-15 times a day to
production.

Does this stuff prevent regressions, lower issues making into production, and
make rolling back simple? Yes. Does it make it infallible? Not a chance.

Pull requests can be first in line to go out the next workday, no reason to
eat into work/life balance and or company events.

------
aljungberg
I think this argument assumes that frequent deploys and good CI/CD is a silver
bullet for _every possible problem with releasing something_.

You could release a well tested new feature with perfect ease and see nominal
stats for hours after, and still have a problem you don’t see.

Maybe your new Calculator Widget 2000 works great for everyone except Big
Enterprise Customer Bob. And maybe Bob doesn’t log on until 7 hours later
because you’re in London and he’s in San Francisco. When he starts up the
Widget, his mixture of data causes a pathological database behaviour. It’s a
performance mistake in your code (that’s hard to find even with rigorous unit
testing and manual QA).

Knock on effects follow like your task workers getting overloaded. So you have
to saunter back in late on a Friday to fix it. You tiredly misdiagnose it as a
usage spike, maybe some big new customer came on. You spin up 20 new worker
nodes. Doesn’t help enough, tasks are executing too slowly. You figure it’s
one of those mysterious intermittent performance issues in GKE. You begin to
upgrade the cluster software in a desperate attempt, your thinking muddled by
too many Friday dinner whiskys, to resolve it through the traditional GKE
incantations. But cluster upgrades take forever and when you finally see the
real problem and roll back your code it’s 2 am and your partner fell asleep on
the couch and your kid wakes up at 6 am and God why did you deploy your well
tested small new feature on a Friday?

------
NotPaidToPost
Any action carries a risk.

It seems over the top to have a blanket freeze but equally risk should always
be considered and all deployments are not equal when it comes to risk.

Deploying Friday evening or during the weekend might actually be a very good
time to do it if that corresponds to a 'quiet period' for your users.

~~~
NotPaidToPost
I find telling when people say they don't deploy because on the potential
impact on the team.

What about the impact on your customers? That should be your priority.

------
onion2k
When I'm looking for a new job I quite like to ask " _Do you deploy on
Fridays?_ " when it comes to the "any questions?" part of the interview
process.

Obviously it doesn't really work for companies that do continuous deployment,
but for those that don't it provides great insight. Often you'll here an
interviewer praise the fact the company doesn't do Friday deploys "because if
it goes wrong that could eat in to our developer's weekends, and we like our
developers so we'd never do that". That's the wrong answer. The right answer
is "Of course, we trust our deploy process."

~~~
locopati
You can trust your deploy process and still deploy bad code because sometimes
something gets missed in design, review, or testing. Why buy that headache for
what is probably a marginal gain? (no matter how much we want to think we're
changing the world.... that deploy is probably marginal gain that can wait
until Monday)

------
33degrees
I worked at a company that not only had Friday deploy freezes, but also the
other days after 2pm, to avoid midweek deploys staying broken overnight. But
some of our repos were set up to automatically deploy to production when code
was merged into master, and didn’t have a develop branch; which meant that
during half your work hours, you were not allowed to merge your pull requests
unless it was for an emergency.

