

Deploying at GitHub  - samps
https://github.com/blog/1241-deploying-at-github

======
cameronh90
How often does code go through security audits? Is every feature audited prior
to deploying to live? GitHub is making money by selling private repositories
which will often contain very sensitive code, so ensuring nobody can gain
unauthorised access to them is presumably one of the top concerns.

I'm interested in seeing how tight security requirements fit in with this
almost continuous deployment strategy.

~~~
notJim
We do continuous deployment at Etsy[1] as well. Here is a talk from our head
of application security (probably not his official title, fyi) about some of
the stuff we do: [http://www.slideshare.net/zanelackey/effective-approaches-
to...](http://www.slideshare.net/zanelackey/effective-approaches-to-web-
application-security).

[1]: <http://www.etsy.com>, the world's marketplace for handmade and vintage
goods.

------
sonnenkiste
Interesting, I saw something like this by looking at the
<https://github.com/mozilla/pdf.js/> project. Take a look at the closed pull
requests. They have some bots listening for commands in comments and do stuff
like unit testing and previewing. The result gets posted as comment from the
responsible bot. Another one is checking master branch for changes and
automatically builds and pushes at gh-pages. Seems to work very well, but
don't know how they build/did it.

~~~
arturadib
Hi Artur from PDF.js here. We built Bot.io to handle the workflow you
mentioned:

<http://github.com/arturadib/botio>

It's written in Node and is trivial to install/use in any Github project.

------
pbiggar
If you're looking for a less complex model of this, you should try our
Continuous Integration and deployment service: <https://circleci.com>. Over
time, we'll be providing the sort of complexity that GitHub provides here, now
we do about 70% of it.

~~~
bostonvaulter2
Looks interesting. Is it free? It seems like it probably is but I'm not sure.
Also when it leaves beta how much will it cost? I don't want to end up
depending on something I can't afford.

~~~
pbiggar
14 day free trial, then $19 for 1 project, $49 for 10 projects, and $149 to
run tests twice as fast.

Feedback on anything, including pricing, very welcome.

~~~
alexchamberlain
per... build? day? week? month? year? lifetime?

~~~
pbiggar
Month! Sorry, I thought that would be assumed.

------
gbin
I might be wrong but for me this is almost a [Hack] -> [Prod] methodology...

Roll back in 30 seconds, cool but how do you manage data / schema migrations ?
You have a snapshot also to rollback any data corruption the last hacking
session could have introduced ?

~~~
technoweenie
Data migrations are done carefully with the Large Hadron Migrator Ruby gem:
<https://github.com/soundcloud/large-hadron-migrator>. Facebook has a similar
tool: <https://www.facebook.com/note.php?note_id=430801045932>

------
amccloud
If GitHub uses GitHub to deploy GitHub, what happens when GitHub goes down?

~~~
johns
This is a guess, but I'm assuming they use the same software running in an
independent environment just for them.

------
wamatt
Encouraging to know this model scales to 100 employees at least.

Purely out of intellectual interest, I wonder if a company the size of Google
or Facebook could also ship in this way, or if the whole release manager/team
is essential.

~~~
jrockway
Sarbanes-Oxley puts a big damper on production deployments at big companies. I
don't fully understand it so I won't try to explain it.

(I will complain though: the law says developers shouldn't have control over
production systems. If that's a requirement, who's going to write the
software?)

~~~
tomjakubowski
Wait, what? Could someone elaborate on S-O restricting developer control over
production systems?

~~~
cam-
Everyone just 'knows' what is in Sarbanes Oxley but when you ask them to point
it out to you in the legislation they cannot find what they were so certain
about 2 minutes prior. We have compliance people and auditors are always
coming in, but when someone claims something is required for Sox compliance,
challenge them on it as 99% of the time it is a convention because someone
told them, or they did it like that somewhere else once, rather then what is
required by law. At the least it will make them justify the
compliance/overhead they are causing you to do as an engineer.

Here is the legislation if you want to read through it or use it to challenge
someone's assumptions about the Sarbanes-Oxley;
<http://www.sec.gov/about/laws/soa2002.pdf>

------
alexchamberlain
I know this is Github, but... Rather than use the Github API, wouldn't it be
more efficient to interact with Git directly? Libgit2 maybe?

~~~
technoweenie
Dogfooding aside, the vast majority of the time is spent running tests and
actually deploying the code. The time to hit the API to merge the commit is
negligible in comparison.

Also, Janky and Heaven are both tiny apps that don't necessarily have access
to the file servers.

~~~
alexchamberlain
But they are hitting the API to get the OID of the master ref as well?

Not having access to the file system is a fair excuse...

~~~
technoweenie
The merge API takes branches: <http://developer.github.com/v3/repos/merging/>

------
tzaman
_WOW_. 175 deploys in one day?

~~~
46Bit
I'd love to hear the backstory behind this. Company hackday, or a day spent
shipping a major set of new features?

~~~
technoweenie
We don't do "company hack days". If you feel like hacking on something, hack
on it.

We do have days where multiple people will be waiting in line waiting for
their chance to deploy their tweak.

That particular day consisted of staff deploys on multiple in-progress
branches, some performance tuning, bug fixes, etc. Nothing crazy.

I'm also quite sure the number counts deploys across all of our applications.
For instance, deploying a change to github-services counts as two, since I
have to deploy changes to GitHub.com also.

~~~
46Bit
Thanks for this, enjoy hearing about Github as a company.

> I'm also quite sure the number counts deploys across all of our
> applications. For instance, deploying a change to github-services counts as
> two, since I have to deploy changes to GitHub.com also. That might explain a
> lot. Still a lot of deploys, but a more sane count :-)

------
smg
How do you deal with the github enterprise version of your software? Does it
have a separate QA cycle? How often do you ship new releases of that?

I am hoping that Github could shed more light on the how they ship an
enterprise version along with the SAASy web version that we all know and love.

~~~
calavera
Yes, the testing/deployment cycle for Enterprise is totally different. We
usually release a major version with new features every two/three months, and
2 or 3 minor versions with bug fixes in between.

We always keep the version of github synchronized with master for
development/testing, although we only release master directly in major
releases. For minor releases we avoid to include major features from github to
keep it as much stable as possible.

------
trustfundbaby
Interesting ... what's your QA process? Do you have a staging environment
where you try stuff out (looking for bugs) before pushing to production?

~~~
psadauskas
We have a staging environment, but its really only used for really big changes
that might need to be experimented with before being deployed. We can also
deploy a branch to a single front end to observe how it behaves with a subset
of the traffic, and roll it back quickly if needed. Also, most large user-
facing features are released as "staff-only" first, so we as GitHub users are
able to play around with it for a few days or weeks before enabling it for
everyone.

~~~
vrish88
How do you guys release features as "staff-only"? Do you have some internal
tool that manages that?

~~~
rtomayko
No. There's simple conventions for adding feature flags
(user.some_feature_enabled?). Features are enabled and disabled by changing
the code and deploying. This works because deploying new code is fast.

