

At what time do you deploy to production? - kirillzubovsky
https://scoutzie.com/blog/at-what-time-to-deploy-code-to-production

======
aytekin
All the time. We do continuous deployment.

The secret is in the tests. Every time you break the site you add a new test
to make sure it never happens again. As long as you run the tests on parallel
you can add virtually unlimited number of tests.

------
danudey
We push to production constantly throughout the day, whenever a new feature,
fix, or update is finished and the developer is confident that it works (and
the unit tests pass). We have a system set up to do one-click (well, four-
click) deployments to all of our servers, plus rolling graceful restarts and
point-in-time rollbacks or specific-commit deploys.

We do double-digit deploys daily and as we get more developers that number
increases. We did 30 deploys yesterday, and >300 in September, and the amount
of time, trouble, and frustration it's saved us is immeasurable.

~~~
kirillzubovsky
Sounds like a really great processes and I bet it was worth investing in. When
we get to the scale of having enough dev power to do this, while also
innovating on the core, certainly! Thanks for sharing.

------
ameoba
4:45pm on a Friday before a long weekend.

~~~
kirillzubovsky
Ahh. the long weekend! Too bad those don't happen too often. We tend to push a
new update every couple of days, sometimes a couple times a day.

------
semerda
You don't need "sophisticated testing frameworks" to validate code quality.
TDD should be caked into your culture. Even if it's not as detailed as some
larger companies.

Pushing code to production should be an automated task that anyone can setup
these days using something as simple as Fabric. Heck, code in reverse release
scripts too.

And if you have few boxes in your farm/lb then push to half of them, monitor
and if all good push to the rest. You should have 0 downtime in this model.
Even with 2 instances under a load balancer.

All this requires some initial time expenditure but over the long term saves
you a ton of time, headaches, lost sleep etc and allows you to push at anytime
of the day.

~~~
kirillzubovsky
I definitely agree on the long-term benefits. At the moment, I think we are
still changing too many things too fast to keep a non-core set of tests. We
are actually using our friends from RainforestQA to do a bulk of testing,
which has been really helpful, but some things still fall through the cracks.
Like an ajax script failing due to a new caching implementation, that only
starts to fail after enough events have taken place. That's when it's really
helpful to know that a large number of your users is offline, and they are not
all going to trip up at the same time.

------
jtbigwoo
Two big companies I've worked have generally followed the same progression of
deployment times:

Stage One: Right after work on Friday (until some big client that files all
their payments on Friday afternoon complains.)

Stage Two: Late on Friday night (until problems arise and all but one of the
mainframe admins have gone camping for the weekend.)

Stage Three: Late on Thursday night (until problems arise the next morning and
half of the DBA's are on vacation and the other half are "working from home")

Stage Four: Late on Wednesday night (until problems arise that take the system
down for 18 hours and impact clients on Thursday morning.)

Stage Five: Right after work on Friday.

~~~
danudey
I worked at a company that had a policy of no changes to production on
Fridays, unless our service was completely down and we were losing people's
money.

Given that it was the sort of place where one developer didn't have access to
deploy code and had to go walk a senior developer through making the changes
by hand (and we didn't use a revision control system), that was probably an
optimal circumstance.

------
abentspoon
Sorry I don't have anything more constructive to say, but you have Europe and
Asia reversed.

5:00 PT is midnight UTC, so Europe is more likely to have just gone to bed.

------
dialtone
The best time to push is the middle of the day of any weekday, possibly when
there's a lot of traffic. You'll immediately figure out if there are problems
and should already have the setup to revert or have enough tests to know that
the basic functionality will be working.

If you push at any other time you'll end up not having any power on finding
out when stuff is wrong. Friday or weekend pushes are an absolute bad idea
most of the time.

------
dsl
Good analytics will help you define what time of the day (or day of the week)
is your traffic slump.

