

Deploying branches to GitHub.com - tmm1
http://githubengineering.com/deploying-branches-to-github-com/

======
100k
I really enjoy reading how organizations have implemented continuous
deployment.

One question I have which is not addressed by the article is how to deal with
database changes. Every database has difficulty with schema migrations to one
degree or another, but MySQL (which IIRC is what GitHub uses) is particularly
bad. In my organization, we are VERY careful with any deploy that contains a
migration.

(I suppose this is where GitHub's staging environments with real world
datasets come in.)

One workaround I've considered is automatically deploying code that doesn't
contain a migration (which is the vast majority) and forcing a more manual
approach to database migrations, to make sure people are on hand to roll it
back if necessary.

~~~
samlambert
At GitHub we take a fairly manual approach to migrations, basically the
migration is done in its own PR ahead of the PR that has the code.

~~~
tonyhb
That's awesome! Seems like a really solid setup. Bufferapp took a page from
your book and have the same deploy setup with hubot... it works out really
well for them, too.

I have three questions to ask about DB migrations (which I can guess the
answers to but would love to hear directly), if that's okay:

How do you handle a DB migration with a staged rollout (two 2 of N production
servers)?

How do you organise timing between a migration deploy and code deploy if one
is done before the other?

How do you handle rollbacks?

~~~
technoweenie
Migrations stay pretty small. Most of them just add a column or index. We also
have to make sure tables aren't too hot to migrate. See a previous post for
more on that: [http://githubengineering.com/using-mysql-performance-
schema-...](http://githubengineering.com/using-mysql-performance-schema-for-
workload-analysis/)

It's also really important that migrations don't affect the running app code.
New columns shouldn't be used yet. Removed columns or tables need to have all
references removed first before running the migration. We confirm this with a
deprecation helper that sends metrics to Graphite.

That's about all I can answer from the app side :)

~~~
tonyhb
Sweet, that makes sense. Thanks for answering!

Was assuming that a column rename would duplicate the column during the first
migration — so that the old and new codebase would work correctly. I guess the
only complication is that you need to keep track of which branches have been
successfully rebased/merged to master so you can run the second cleanup
migrations.

Thanks again!

------
mpdehaan2
The 'merged once deployed to production' thing, yes, I know even if advocated
by GitHub, seems extremely weird to me. It does seem they have a staging check
first, which is good.

It seems you'd want to merge it first, so that you know it when merged with
"all the things" on master, so it more closely mirrors what you are going to
get once it's merged in.

So they could just merge first, and then if staging passes in their CI system,
automatically deploy to prod, which is the way many orgs do it.

My point is though, you'd want to deal with the merge fun (if any) first, else
you are deciding to test branches (pull requests) that only have ALL of the
commits from master (rebased, etc), so it's easier to just make sure they hop
on master first, else you might "remove" something from prod for a while until
it's merged in. Not good.

They may have some things to deal with that, but in this case, it doesn't seem
like something I'd recommend for most people, and feels weird and organically
evolved. One branch may not have the commits another has and both could be
deployed without merging, leaving the github deployed code state fluctuating
back and forth as one commit drops out and another drops in, before finally
both are in at the same time.

I wonder how this is handled?

~~~
Lukeas14
I'm guessing that the master branch is first merged down to their develop
branch or directly to their release branch. That way their release branch does
have everything in master already.

~~~
daigoba66
Indeed that is the case:

> Since master is our stable release branch, we want to ensure that any
> branches being deployed are caught up with the latest code in master. Before
> proceeding with a deployment, Hubot will detect if our branch is behind and
> automatically merge in master if required.

------
techpeace
If one of your developer's Campfire (assuming y'all still use this) accounts
gets popped, does the attacker now have the ability to deploy to production,
or is there some other mitigating factor not mentioned here?

~~~
sciurus
I'd be surprised if they don't have additional security. Here's some info on
what Box did to add authentication and authorization to Hubot commands.

[https://cloud.app.box.com/dodatx2015](https://cloud.app.box.com/dodatx2015)

~~~
techpeace
Thanks! This is exactly what I was looking for.

------
oliverzheng
If I read this correctly, only 1 dev can test on production at a time. How
does this scale as the company grows?

~~~
joelthelion
I suppose you could have a hierarchy, with team leads pulling from their devs
and deploying the whole team's stuff when it's ready.

~~~
juliangregorian
That would defeat one of their major wins, ownership/agency/responsibility.

------
cdnsteve
"spider-skull-island" can I get access?

~~~
technoweenie
That's just a special staging server for Venture Brothers nerds (no one else
wants to type all that out).

------
jontro
We're following github flow and are currently doing these steps manually on
deployment (i.e. merge branch with master, check out branch). Then roll back
to master if something fails, otherwise merge master with the branch. It would
be great with some tooling to help this. Has someone done open source work on
this already?

~~~
samlambert
[https://hubot.github.com](https://hubot.github.com) and
[https://github.com/atmos/heaven](https://github.com/atmos/heaven).

------
earless1
I've been trying so hard to move my organization towards a chatops approach.
We are making progress, it's just slower than I want.

------
ing33k
Deploy Locks looks interesting. does any one has more info on how this can be
achieved in a git pull based deploy flow ?

