
How to Deploy Software - jmduke
https://zachholman.com/posts/deploying-software
======
lucaspiller
One thing this glances over is that you should have something monitoring your
production systems to make sure that they are running correctly.

To start with get something to monitor errors/exceptions and email you. To
name a few services:

[https://airbrake.io/](https://airbrake.io/)

[https://rollbar.com/](https://rollbar.com/)

[https://github.com/errbit/errbit](https://github.com/errbit/errbit) (can be
hosted on Heroku for free)

Also make sure that you have accessible logs that log useful information
(timestamps, the user making the request, unique request ID). Then use syslog
or a SaaS service to aggregate logs from all servers in one place, and keep
them for as long as you can.

~~~
blister
Shameless plug, but my co-founder and I are about to launch our powerful
logging service this month. One of our features is session-specific logs and
the ability to send alerts at user-defined thresholds.

We'd love to get some feedback on our product when we launch to beta, so if
you're interested, please sign up for our email list at
[http://logdebug.com](http://logdebug.com)

~~~
joshmanders
One suggestion is remove the "Most Popular" banner above Small Business on
your pricing. Being as your site is plastered with under construction, it's a
turnoff seeing such false advertisement.

~~~
blister
Sure, good tip.

Though the pedantic butt-head in me wants to point out that out of all of our
paying customers, they're all using that level of service. :) So
technically...

~~~
johns
So keep it. Ditch the under construction stuff instead if you have customers.

------
paulstovell
I really enjoyed this article. As an industry, when it comes to something
essential like source control, we seem to have converged to a common set of
practices and workflows. Deployment is arguably just as important, but I think
the practices are very different on different teams. This article is like a
more practical version of Continuous Delivery.

Three areas that I think would have been worth including:

1\. Pre-production.

You deploy to test & other pre-prod environments more often than prod. They
should use the same scripts/tools/processes as production deployments, only
with different permissions.

2\. Configuration.

Test and production environments will always have different config settings,
so no team will ever be able to deploy to more than one environment without
encountering this problem. I think there's still an open question around
whether those configuration settings should live in the same source control as
the code, in a different source control repository, or a dedicated system.
Source control systems and sensitive values (passwords, API keys, etc.) don't
always mix.

3\. Build your binaries once.

The article is more focussed on dynamic languages, but for compiled languages,
I think this is important. If you branch, compile, deploy to test, test it,
get the all clear, then compile again and deploy to production, there's a lot
of opportunity for differences between what you tested and what goes to
production to sneak in.

In fact even for dynamic languages, this might be a valuable practice. What if
the JS minifier on one build server is different to another, and the deployed
script ends up being different in production to what was tested.

Disclaimer: I'm the founder of Octopus Deploy, and these practices might be
biased towards enterprisey .NET/on-premises deployments rather than cloud
hosted, dynamic language projects.

~~~
Spearchucker
_1\. Pre-production_

Pre-production should test production in a non-production environment.
Specifically, if a system connects to, say, SalesForce, and you've been using
a local sandboxed instance of SalesForce in dev and UAT, the pre-prod should
connect to the real, live SalesForce. I have a pre-prod environment for my
current project, and it's useless because org policy states that only prod
environments can connect to live instances of anything. Completely invalidates
the reason for having pre-prod.

 _2\. Configuration ...will always have different config settings..._

Yes, aside from pre-prod. Pre-prod and prod MUST be mirror images, INCLUDING
config settings. Identical. The same in every way except for host names.

~~~
UK-AL
We use octopus deploy(Great software btw), most people use configuration stuff
mostly for connection strings. Everything else is the same.

------
kelvin0
In theory, feature flags seem like a good idea. Until you reach a point where
too many flags become difficult to test in an exponential tree of
combinations. Also, it demands tight discipline to make sure each new flag
properly isolates some new feature ... Has someone had success with this idea,
and be able to 'tame' an explosion of flags in their codebase? really curious

~~~
yazaddaruvala
On my team of about 10 devs, we create / destroy about 5-8 feature flags a
month.

The only thing you have to remember is: Its either going to production, or its
getting dropped.

There is an expiry date built into the feature flags framework (90 days), and
either they need to be explicitly cleaned up and deprecated, or extended with
a valid reason attached.

Also, we code review every commit. If any diff is coding around/through a
feature flag, the first question is always: "Can you clean up that experiment
first?"

~~~
ronnier
Sounds very familiar. Are you in Seattle?

~~~
yazaddaruvala
Something like that, yeah

------
blowski
> Deploying major new features to production should be as easy as starting a
> flamewar on Hacker News about spaces versus tabs

Great writing. Spaces all the way.

~~~
felixmc
Would someone mind explaining the argument for spaces over tabs? Possibly
naive young dev here.

~~~
ktRolster
How many spaces do you like to indent your code? Some people like 3 space
indentations, some like 8 spaces, and some like 5. It's very personal (but
anyone who doesn't use 3 is a reeky brazen-faced mammet).

A simple solution is to use tabs, because then every developer can set the tab
distance however they like, and they are happy. It breaks down when you have
code that is aligned beyond the indent, like this:

    
    
      if(a && b &&
         c && d &&
         e && f   ) {
    

In that case, increasing (or decreasing) the tab distance will ruin the
intention.

The best solution is to use tabs to the point of indentation, and then spaces
thereafter, but a lot of code editors don't support that, so in practice it's
hard to implement, so people use spaces to preserve their formatting when it
gets uploaded to github.

~~~
collyw
Mixing sounds like the worst option.Its what has cause horrendous looking code
in the past when you open someone elses code and they are mixed.

Python and spaces for me.

~~~
ktRolster

      > Python and spaces for me.

Sounds like you are a very open-minded individual. Not.

~~~
collyw
I say this from experience, not narrow mindedness.

Open a file where it is all spaces, and it looks the same on every machine.
Open a file with a mix of spaces and tabs, and it often turns out an absolute
mess.

------
rb808
Does anyone else work on systems that take 3 hours to back up the DB, an hour
to deploy, 1 hour to start up and a few hours for users to check out
functionality before business opens on Monday? No to mention the federal
regulations about what paperwork is required and who can even access
production. Maybe I'm in the wrong site.

~~~
RandomChance
Hah, we (hopefully!) get an update window every 18 months! And a staging a new
production like test environment starts with another $100K invoice..

And thank goodness Microsoft hard-coded in a 10 hour delay to ensure the KDS
root key of a domain is propagated, even if I create it before I create
additional domain controllers...

Such fun!

~~~
tumba
Off topic, but to create an immediately effective KDS root key, just set the
effective time ten hours in the past. You can validate propagation by looking
for the 4004 event in the KDS event log. This is probably not a good idea in
production, but is useful when building/rebuilding a lab.

    
    
      Add-KdsRootKey –EffectiveTime ((get-date).addhours(-10))
    

See [https://technet.microsoft.com/en-
us/library/jj128430.aspx](https://technet.microsoft.com/en-
us/library/jj128430.aspx)

------
ngrilly
The post is interesting, but it doesn't mention two major difficulties: zero
downtime deploys and database migrations.

~~~
herge
We use django migrations, and I still haven't found a way to do zero downtime
deploys and migrations short of doing the following:

If we are trying to deploy migration #1, first deploy a version B of the code
that supports (but provides the same set of features as before) the db both
after migration #1 and before migration #1 (maybe with the help of a feature
flag that is set in the migration). Do the migration. The deploy a version C
of the code which removed the feature check above. But all this requires 2
different versions of the code and a lot of process just to ship out one
migration. It gets combinatorically worse if you have more than one migration
to deploy.

~~~
raziel2p
I wish there was a way to "pause" incoming requests in web servers. Most
deployments (migrations + code) take less than a few seconds, and I'd be fine
with some users having to wait 2 seconds for a request to finish over their
request hitting a 500 (due to inconsistent code/database) or 503 (putting the
site into maintenance mode).

~~~
wvenable
Usually we aren't deploying a schema change that's really huge so we just go
for it and let the application crash for those users who happen to hit a place
where the code/schema are out of sync.

Zero (no crash) downtime deployments seems like too much effort for too little
gain.

~~~
AznHisoka
Agreed. Seems like a case of optimization for technical completeness rather
than a business need.

------
UK-AL
We have something similar. But I don't like branches, I prefer single trunk
development(generally agree with Martin fowler) + feature switches to isolate
wip features. We store all binaries built. so we just roll back to the
previous binary, which is a single button click for us.

~~~
Jemaclus
Curious... Why don't you like branches?

~~~
UK-AL
[http://martinfowler.com/bliki/FeatureBranch.html](http://martinfowler.com/bliki/FeatureBranch.html)

It's a controversial article though.

~~~
harlanji
I'm pushing for toggles on my team, haven't made a single feature branch in
this repo resulting in no merge conflicts (yay).

Sofar I'm the only one regularly doing it without feature branches. Running
with the idea that just because you can branch cheap doesn't mean you should.
Of course toggles are technical debt to be managed, but so are branches.

I've found good practice has been mentioning which toggles are available on
the README with their defaults (could be generated)... they should be tracked
and removed ASAP. I read a newer article that breaks them down into categories
[http://www.infoq.com/news/2016/02/featuretoggles](http://www.infoq.com/news/2016/02/featuretoggles).
Toggles over branches are showing value as we run different variants on
staging without having to redeploy different builds, but instead changing a
launch variable. It's especially clear with 2+ WIP features. We're using
environ with Clojure which doesn't have any fancy runtime toggling, but that'd
be another thing to look at.

~~~
falcolas
The tech debt associated with branches disappears with the completion of the
merge request. The tech debt with feature flags doesn't disappear until
someone gets around to removing the (now) dead code.

Personally, I use both, based on what is best for the feature. Why not, after
all?

~~~
harlanji
I love branches in theory, but the main problem has been in practice around
CI. It's a nightmare trying to manage a staging environment with feature
branches involved... given better tools/a better build pipeline I'd go back. I
agree, toggles are basically programs branches compiled in... analogous goto
vs if/else control structure.

But day-to-day, stakeholders want to test feature X that they've heard is
going well but it's not stable enough for develop? Okay, let's [engineering
time and $$$]. vs. adding FEATURE_NEWDB=true to the upstart script. We've
already got automation engineers helping us out, but until the deployment
problem is solved toggles are more practical in our case.

~~~
falcolas
How hard is it to set your CI to run off a separate branch? Or to run CI for
all branches which are checked into a repo (a common option I've seen
implemented)

------
nissimk
Nobody said it here yet, so I just wanted to mention that the design of this
blog is really nice. I love the font, links, blockquotes and chapter title
images.

It also rendered equally nicely on my android phone and my desktop browser.

I saw that the author has a github repo for an older blog style for jekyl, but
I'd like to see a similar thing for this one.

Thanks

~~~
holman
Thanks! Was thinking of open sourcing this soon as a one-off. I'll see what I
can do. :)

------
TickleSteve
"How to deploy _web_ software".

The software world > web servers.

(Sorry to be picky, but some people seem to assume that all software is
developed for the web these days whereas the web world is just a significant &
vocal minority).

~~~
collyw
A great deal of software is web based these days. You install apps on your
phone or your desktop. You deploy to a server.

~~~
TickleSteve
true, but its still a 'sizeable minority'. As far as number of deployments go,
the embedded software world massively outweighs server, client, mobile and
desktop.

Unfortunately (or fortunately), we in the embedded software world tend to be
less vocal than the others.

~~~
collyw
I wouldn't have though deploy was the correct word for embedded.

If its one piece of software I would think of it as an install. If it's
coordinating multiple pieces its a deployment.

------
oebs
Thanks for the writeup - very helpful. It's always good to get a view of how
others are solving the same problems oneself has.

That said, the article does come off a bit as trying to be authoritative, but
at the same time it doesn't leave enough room for possibilities where
alternative approaches may have merit as well (i.e. "this is how to do it" vs.
"what worked well for us, ymmv"). Newbies that read this article will think
that the principles described are the canonical way and even try and apply
them in scenarios where alternatives may prove superior.

Other than that, a lot of good advice, well done!

------
AlexTes
I've been arguing for using Git Flow. Reading the post I have to say the stand
our lead takes against Git Flow and in favour of very cosy CI is perhaos
steonger than I realised.

He argues for pushing about as often as possible. With our small team thats
very do-able, every push gets tested and linted by the 'blue' or 'green'.
You're supossed to only push passing code which you easily can by running the
tests and lint locally. So instead of all the pain points mentioned in the
post you write passing code, pull and rebase on other passing code, and then
push. Little code review, no worries about hasty reverting, few / early
conflicts keeping us from trilling each other up or writing incompatible
features.

The reason I argue for Git Flow? Our tree is an absolute mess. Most often a
single chain, of often linearly scrambled features. In other words removing
one feature would be hard and require a bunch of legwork, not a couple Git
commands.

If anyone strongly feels there's a better way for a small team than lightning
fast CI let me know!

------
doublerebel
I wish native Nodejs deployment was a solved problem, but there really is no
comprehensive and universally used tool for deploying Nodejs using Nodejs.
ShipIt mentioned in the article is barely a year old, it has a short
featurelist and short list of users. PM2 (Keymetrics) is not bad but is buggy,
also they seem a bit overwhelmed at the moment. Flightplan is decent but the
syntax is more awkward than ShipIt. Every other common language has a stable
deployment tool besides Nodejs.

I ended up going with Distelli, it's a SaaS but it's fantastic. These days
deploys often involve more than just one app or language, and I really prefer
a tool that can ship anything. Also, having a GUI to see deployment statuses
is invaluable. With those requirements none of the Nodejs tools can stand up
to the other, more mature utilities. And rather than have to write all my
deploy logic in another language, I just purchase the service.

~~~
supster
What issues have you run into with PM2? I'm running it in production myself,
so I'm curious.

~~~
hodoublesy
I've really enjoyed working with PM2 in production for the last 4 months...
although our ramp up has been slow.

------
datr
I'd be interested in hearing people's thoughts on deploying feature branches
to production before merging them. I've generally followed more of a git-flow
approach [1]. This seems to have the advantage that multiple feature branches
can be grouped and deployed together - thus, avoiding the problem in the
article of the deploy queue becoming a bottle neck.

[1] [http://nvie.com/posts/a-successful-git-branching-
model/#crea...](http://nvie.com/posts/a-successful-git-branching-
model/#creating-a-release-branch)

~~~
ikawe
> multiple feature branches can be grouped and deployed together

I would respectfully argue that this is not a desirable feature.

Although you might save some time by deploying multiple branches at once, you
cloud the waters of what and how to roll back.

I think a better idea is to make deploys easy, quick, and revertible so that
you can deploy early and deploy often, and in the event of a rollback, you can
rollback just the broken feature.

~~~
datr
That's true but it would seem like there would be other disadvantages to.

For example, when do you do testing? If we test as soon as the pull request is
opened, we know that master is going to change a lot between now and when we
finally deploy this code so the tests might not be valid.

If, on the other hand, we wait to test just before we deploy to live we risk
locking up the queue for too long. This might not be an issue if you're test
only take a couple of minutes to run but if you have lots of integration tests
(like facebook for example [1]) then it could become a big issue.

Is the solution to this, you just accept that the codebase you're testing
won't exactly be the same as what's deployed to production and the risk that
comes with that?

[1] [https://developers.facebooklive.com/videos/561/big-code-
deve...](https://developers.facebooklive.com/videos/561/big-code-developer-
infrastructure-at-facebook-s-scale)

------
platz
Why worry about what is on master, if you save your build artifacts. so that
if you need to go back to a the previous behavior, just redeploy the previous
_productuon build output_

~~~
p4lindromica
You code is written in golang. You've been compiling it with golang 1.5.1.
Then golang 1.5.3 comes out with critical security fixes for their TLS code.

That is why you care what is on master: because you need to rebuild if your
runtime changes.

~~~
joneholland
We deploy straight from master, but tag each build.

If you need to rebuild a specific version, it's as easy as checking out the
tag.

~~~
softawre
This feels like the correct strategy to me (and it's what we do too).
Deploying many branches to production seems like a nightmare at any scale.

------
nathancope
Interesting article. Frank and honest. Good to read the experiences of others.

To get the disclaimer out of the way I'm a co-founder of Vamos Deploy. Our
product addresses many of the deployment problems that have been discussed
here so I thought I'd mention it. We are looking for feedback on the product
and an early adopter or two -
[https://vamosdeploy.com](https://vamosdeploy.com)

I'd like to cover some techie details here. Vamos Deploy encapsulates an
application with it's dependancies and runtime config so it can be deployed as
one to any number of machines irrespective of the OS (well, Linux and Windows
at the moment). This encapsulation is achieved by configuring a 'grid' with
all the application package versions, library/runtime dependancies, runtime
property values and local repository names (hostnames usually). When the grid
is deployed (all via CLI) the respective local repos get updated. You can have
multiple grids on a host (in a local repo) thus enabling multiple, differently
configured, encapsulated applications that don't conflict. It avoids
duplication by the grid sharing the underlying application packages and
libraries in the local repo. There is a audit log of all actions for
traceability and transparency. A simple ownership model prevents non-prod code
getting into production and restricts who can deploy to production. It can be
combined quite easily with any config management tool for release
orchestration. You don't need RPM/Deb packages or deal with Yum repos. We have
concentrated on making it easy to learn and use so max benefit can be attained
quickly.

I'll stop there. Be interested in anyones feedback here or
[https://vamosdeploy.com#contact](https://vamosdeploy.com#contact) for a chat.

------
richardwhiuk
The title of the article was "How to Deploy Software", but almost all of the
advice only works for server side software where you have total control over
the deployment environment.

I'd be much more interested to learn about how people develop mobile and web
apps, where feature flags are far less useful as you need to push the entire
app to the AppStore, so your iteration time is much slower.

------
LeZuse
I also recommend Zach's presensation [https://speakerdeck.com/holman/how-
github-uses-github-to-bui...](https://speakerdeck.com/holman/how-github-uses-
github-to-build-github) [selfpromo:] When we were thinking about deploying our
frontend builds, we got inspired by Ember's CLI Deploy pipeline
([http://blog.firstiwaslike.com/deploying-ember-cli-
apps/](http://blog.firstiwaslike.com/deploying-ember-cli-apps/)) and we've
built something similar for our Webpack based app
([https://github.com/productboard/webpack-
deploy](https://github.com/productboard/webpack-deploy)). Together with Git
flow methodology we basically removed all friction from deploying new
versions. Would love to hear your thoughts!

------
agentgt
I don't know if I agree with the branch on every deploy particularly if you
have a small team and use Mercurial where named branches live forever. I wish
the article discussed dependency management more.

Instead of branching we tag every deploy and use dependency management heavily
(ie maven, npm, etc). That is the project that gets deployed never really has
any branches but is composed of lots of smaller projects each in their own
repository which may have branches but they have to be released.

This approach cuts build time, improves coupling/cohesion as well as
facilitate a possible transition of OSS useful components (that do not provide
a competitive advantage nor or proprietary).

I have seen way too many projects that have this giant monolithic source tree
(particularly PHP projects) and thus have to rely on branching much more
heavily. I firmly believe this is the wrong approach.

------
softawre
> When you're ready to deploy your new code, you should always deploy your
> branch before merging. Always.

Does anyone actually do this? This seems counterproductive - what if there are
multiple branches?

~~~
NLips
Assuming you have a product/release branch and many others, then I'm guessing
the author means you should merge product into your own branch and deploy
that, before merging your branch back into product (and deploying it).

That should work fine with multiple branches in most cases, so long as you
have a system to stop anyone else deploying their branch while yours is
running.

------
pippy
I use deploynaut as my deploying tool, and I have to say it's made the process
much smoother. Previously I'd simply use git to update a code base or sql
workbench/pgadmin to update a database.

------
microcolonel
The body text in this article is illegibly thin, please consider moving to
weight 600 so that people can read your text. You've worked hard to write it,
now it is time for people to read it. :-)

~~~
pmontra
I didn't notice, but my NoScript blocked the web font and the default font is
more legible with that color. In both cases changing the color from rgb(100,
100, 121) to rgb(80, 80, 100) is enough to improve the readability of the
text. font-weight: 600 seems a little extreme.

~~~
jorams
For some perhaps the color of the text is the only complaint, but for me the
font is also illegibly thin [1]. I got around it by disabling the web font,
because just setting the font weight to 600 didn't fix the odd shapes [2].

The bold version of the font is also available as a separate font family
(AvenirNextLTW01-Bold). It looks much more like a "normal" font weight and is
incredibly readable [3].

[1]: [http://i.imgur.com/uVHXptR.png](http://i.imgur.com/uVHXptR.png) [2]:
[http://i.imgur.com/IXwR6EU.png](http://i.imgur.com/IXwR6EU.png) [3]:
[http://i.imgur.com/xgz7EtR.png](http://i.imgur.com/xgz7EtR.png)

~~~
pmontra
I noticed that thin fonts are not so thin on high DPI screens. Example: that
font is more readable on my 9" 2500x1600 (approx) tablet than on my 1080p 15"
laptop. Maybe they are designing for retina Macs.

------
sabujp
[http://spinnaker.io](http://spinnaker.io) : deployment software made by
netflix (king of daily prod pushes), google, and the community for aws, gce,
azure, etc

------
gandhineil
shoutout to sublime for development

------
my5thaccount
I just double-click a command file on my desktop and I'm done.

