
How GitHub No Longer Works - bado
http://zachholman.com/talk/how-github-no-longer-works/
======
carbon8
I've found it interesting how many people in the Bay Area, particularly
veterans of the 90s boom, discount a lot of the ideas pushed by GitHub and
37signals. Anedotally, one of the common responses whenever I reference either
company is, "Yeah, but how big is [GitHub|37signals], really?" It's actually
somewhat surprising to me how consistently I've encountered this response (so
consistent, it feels as if it was distributed like a political talking point).
As mentioned in the slides, publicly announced funding definitely serves as a
signal, and I haven't heard anyone say that about GitHub since then, though
someone said it to me regarding 37signals within the past few months.

What I think it really has to do with is that GitHub seems to make an effort
to hire doers, whereas many companies develop processes to deal with non-
doers, eg, having staging servers so non-technical staff can check the work of
"their" developers. Remote work also doesn't serve the interests of non-doers,
since they want to Have Meetings and Make Decisions.

~~~
runako
>> staging servers so non-technical staff can check the work of "their"
developers

Surely staging servers have some other benefits? Maybe some teams consider
running code on the production setup for the first time to be risky to their
customers? Perhaps some teams have encountered differences between dev laptops
and production servers, which run on different OS/memory/etc. combinations?

~~~
carbon8
Short-lived staging environments (eg, a temporary clone of a production
environment) certainly have a place, such as when making architectural
changes, but these kinds of changes are generally not happening on a regular
basis if you are making incremental changes and doing continuous deployment.

I'm sure that there are companies that have a valid engineering need for
perpetual staging environments rather than feature flags, but I've seen
absolutely no evidence that staging servers are commonly engineering driven.
Certainly in every part of the web startup world I've had contact with or
heard about, staging environments have consistently been for product QA in
organizations with heavy-handed processes and/or product management by non-
technical stakeholders.

Edit for the people responding: This is not about haphazardly pushing to
production. You should familiarize yourselves with continuous integration
([http://en.wikipedia.org/wiki/Continuous_integration](http://en.wikipedia.org/wiki/Continuous_integration))
and the various deployment strategies of major web companies.

~~~
diek
You honestly can't think of good engineering reasons not to push new code
directly into production?

~~~
angersock
too busy crushing it to use a staging server

sorry you aren't a technical coder

you probably even file bugs instead of just monkey-patching it in production
like a real man

(srcub)

------
eknkc
I love almost eveything GitHub does, and the way they do them. I'm a GitHub
fanboy. There it is.

I had a similar crush on Google, maybe 6-7 years ago, they lost me somewhere
on the road.

~~~
yapcguy
What's the big deal about Github? Too much hype.

Whenever you hear about the number of repos, most of them are just forks. In
fact, I've created many by accident because Github changed their UI so
clicking the icon instantly starts a fork instead of asking for confirmation
first. I guess it's good for bumping the numbers up when getting private
equity investors interested (cynical, I know).

Security wise, meh. Why would anybody pay to put their closed-source
proprietary code onto Github? More holes than a sieve with their Rails setup.

Ok, you want somewhere to stash some private repos, nothing confidential. So
why would you pay for Github when you can use Bitbucket for unlimited private
repos? Plenty of other providers too.

Their enterprise product? Expensive meh. Atlassian's enterprise Stash is
shared/open source when you get a license. You can also just install Gitlab,
Gitorious, etc. for free on your own servers.

Github fanboys get excited whenever a big name dumps some code onto Github,
like when Linus copied the kernel code over, but it was just a dumping ground
while they sorted out server problems of their own. Ubuntu uses Bazaar and
Trac, Linux Kernel development uses git on their own servers, KDE is over at
Gitorious. Heck, there are some people still using Darcs and SVN.

Github is popular but it's not essential unless crave social karma points and
your name/handle in lights.

Sorry for the rant like writing, lots of thoughts above, but I just never saw
what the big deal about Github was. A good business and a slick offering but
it's not going to suddenly transform your software projects or your abilities
as a programmer.

~~~
techpeace
The big deal about GitHub is that they were in large part responsible for the
current Open Source Renaissance we currently find ourselves in. Do you
remember the world before? Where you would hunt for someone's private SVN repo
on their personal site and email them patches, only to hope that something
comes of your effort? Compare that with the world of today, where, if I'm
using GitHub, I can submit a fix to a project I happen upon entirely from my
web browser, if I wish. The maintainer is notified immediately, we can have a
line-by-line discussion about the change, and then it can be merged - also
entirely from the browser, if that's what you're into.

Introducing DVCS and the pull request model to the wider development world is
the reason that the pace of OS software development has accelerated so much in
the past half decade. Of course, GitHub did not invent git, but they made it
easy to host your repositories, and once Rails migrated to GitHub, the rest of
the Ruby world came with it - followed shortly thereafter by a sizable chunk
of the wider web development community. If you use one of their many
competitors, that's fine - but they wouldn't exist had it not been for GitHub.
Remember that Bitbucket started out as a GitHub clone for Mercurial. Gitlab is
unashamedly an OS clone of GitHub.

> More holes than a sieve with their Rails setup.

This is just unnecessary FUD. The GitHub and Rails teams have both been fast
to deal with security issues as they are raised. Just because a security issue
is well publicized, it does not mean that every other piece of software ever
written does not have its own security issues.

~~~
strumptrumpet
Renaissance? Open source has never been less healthy. All the previous social
constructs around properly documenting, testing, and _releasing_ stable
versions of your code have been swept away.

Instead we're faced with the constant churn of semi-functional code, users
working in silos and then showing up with patches well after it's too late to
give them direction, animated GIFs instead of careful engineering discussion.

Github optimized being lazy, and this was appealing, and in doing so they
broke most of the technical and social structure that held together open
source's ability to produce reliable, stable, well-documented software.

I've watched stable projects either wither and die or become commercialized,
and now instead we live in a world of rolling mostly-broken hacked out
releases (if there are releases at all), a confusion of forks (which one is
the 'real' one?), while animated GIFs and "oh snap" responses permeate our bug
trackers.

This isn't a renaissance, it's a circus.

~~~
benihana
>Github optimized being lazy, and this was appealing, and in doing so they
broke most of the technical and social structure that held together open
source's ability to produce reliable, stable, well-documented software.

And yet, here we are, years after Github optimized being lazy and broke most
of the technical and social structure of open source, and we still have
_hundreds of thousands_ of usable and awesome open source projects available
to use and contribute to. The internet and the apps built on it still run just
fine, people are getting more done with technology than ever before, and more
people are getting their code out to the world than before Github.

I think that you're either overly cynical and negative, or you're upset
because Github has taken away implicit social power you used to have or you're
pissed off cause you have to deal with the plebs who use your obviously
brilliant software in ways that aren't correct (i.e. ways you didn't think)
then have the audacity to file bugs on it in ways that are wrong (i.e. ways
you don't like).

~~~
strumptrumpet
Your core premises are flawed:

1) The internet and the apps built on it still run just fine.

There is very little maintainership and investment in core difficult
technology development. Most core projects that make up the foundation of the
internet subsist on the oft-dwindling maintainership of what you seem to
consider to be a legacy generation of engineers.

Outside of areas where companies hold direct commercial interest, _many_ core
technology projects are withering or stagnate. There are an infinite variety
of new JavaScript frameworks, however.

2) Hundreds of thousands of usable and awesome open source projects available
to use and contribute to

We're reinventing wheels at a prodigious pace, but your comment demonstrates
and underlying shift in the opensource mindset that Github has invoked.
Whereas open source was previously something to be produced as a stable,
reliable entity, and consumed by users, it has instead become an expensive
participatory process for all comers, in which stability and reliability and
even documentation is discarded in favor of quick fixes and local patches and
increased expenses for the entirety of the ecosystem.

3) More people are getting their code out to the world than before Github.

It's the conceit of every generation that they exceed the previous, but this
statement (and the implication that this is due to Github's introduction) is
simply not true.

~~~
chipotle_coyote
You're both arguing by anecdote here. It seems to me self-evident that Github
has many more projects on it -- not just forks of existing projects, but
original projects, period -- than its predecessors and competitors do. This
just about definitionally implies that it has a lot more crap. While it would
be hard to prove that Github has a higher percentage of crap on it than its
predecessors and competitors, that's certainly _plausible._

However, you're essentially taking it as a given that the _absolute number of
worthwhile projects_ has dropped thanks to GitHub. If there's convincing
evidence of this, I'd honestly like to see it, along with a plausible theory
as to why that would be the case. What does, for example, Sourceforge get
right that GitHub doesn't? (At least from _my_ anecdotal experience,
Sourceforge is in fact full of under-documented, unfinished and effectively
abandoned crap to more or less the same degree that GitHub is.)

~~~
strumptrumpet
> _However, you 're essentially taking it as a given that the absolute number
> of worthwhile projects has dropped thanks to GitHub._

Not just dropped -- they're drying up. I can only speak from anecdote (nobody
has paid me to run a study), but while I've seen no dip in usage of my
libraries, and I've seen my projects explode with half-baked forks on Github,
I've seen mailing list participation and worthwhile code patch submissions
drop to very nearly 0.

> _What does, for example, Sourceforge get right that GitHub doesn 't?_

SourceForge essentially died out for modern projects upon the release of
Google Code in 2006.

However, what (traditionally) SourceForge and Google Code did right -- and,
what projects did in their own hosting for decades before and after that --
was place the _project 's_ community in the forefront, and the _code_ in the
background.

This meant that documentation, releases, mailing lists and other constituents
of a vibrant community project were placed in the forefront, with the code
being something that one worked on as _part_ of the community.

By contrast, Github made projects secondary. The code was (originally) always
attached to an individual account name. The primary project page was the code
itself. Forks existed at the same namespace hierarchy as the projects they
forked.

The result was that Github sucked community energy into Github itself, and in
doing so, began to redefine the community social constructs in a way that
allowed users to maximize social and personal rewards while minimizing work
necessary to conform or participate in the _project 's_ community.

~~~
pbowyer
+1. I couldn't agree more.

------
canistr
I'm always skeptical when companies say they have "no managers" and are a
completely flat organization with 100+ employees. I don't pretend to fully
understand organizational theory, but I take it that at some point, there will
be someone doing the "management" work. Whether that's someone with the title
of ninja/hacker/product guy/project manager/director/CEO/BDFL

~~~
ben_straub
(Disclaimer: Hubber.)

Here's the thing about "management work": it turns out you don't necessarily
need management to do it.

Setting priorities? This can be done through consensus.

Making sure schedules are met? Don't have schedules. (1)

Hiring? Have everybody do it.

Giving out raises? Set up a deterministic system and forget about it.

Giving feedback? Everybody can do this.

Running meetings? Don't have meetings.

Communicating with the other parts of the company? We have technology for
this.

1) EDIT: forgot my footnote. Of course, we're in sort of a fortunate situation
here, and not every company can just not have a schedule.

~~~
cpeterso
_> Making sure schedules are met? Don't have schedules. (1)_

Did you forgot a footnote (1)? I'm curious to read more.

~~~
ben_straub
Heh. Whoops. Thanks. :*)

------
adamb_
Please remove "[video]" from the title.

As of right now there's no video (there's a placeholder saying it's coming
soon.)

~~~
fn
That's funny. I didn't click on the link cause I thought it was a video, until
I saw this.

------
timc3
What I get from these really great presentations presentations/posts/slidehows
that Zach creates is that obviously GitHub is an interesting place right now,
and Zach has got some really good insight but I can't help feeling that Zach
really needs to get some experience with working at few more companies for
some of the points he makes. Companies that succeed in different ways,
companies that get bought out, companies that fail.

From the outside GitHub is in a really interesting space in that they can dog-
food their own product very successfully, they have a huge market and
intellectual share, and he writes as if this is the way that all companies
could operate - but they simply can't.

For instances some companies have to play safe with what they say and do
because they are in regulated industries, some just operate by people that
have other values or outlets for their own time. I personally think it's fine
if a company that I use has given up blogging or whatever to spend more time
on creating a great product, or doing that while enjoying friends and family.
I will leave/stop buying when the company no longer creates value for me, not
when they stop talking at events or the original founders cash out because
some people are born to create new interesting companies but are not suited to
the 100+ person growth.

Zach if you are reading this, keep up the good work, but damn your writing is
going to be interesting in 10 years time with a few more companies under your
belt.

------
rschmitty
"Our tech stack shrinks as we age, fewer trendy languages and databases, your
product should be cutting edge, not your tech"

I'm glad this was said. I find it hard to quell the excitement about new FOTM
language/framework/database in our internal team and always wanting to use the
hot new stuff that HN is raving about.

------
bronson
Seems like 20% of that slide deck is about chat rooms... GitHub still uses
Campfire for chat?

I tried Campfire and Hipchat a couple years ago, neither stuck. Not sure why,
maybe poor offline notifications and logging? Or maybe our team didn't have
enough timezone overlap or things to chat about? Hard to say.

Is strong chat as important to Github as Zach implies?

~~~
holman
Absolutely.

As I mentioned, we have 150+ rooms that we chill in, and most of our day-to-
day work practices are tied into chat. Mention someone's name in a room
they're not present in and we send push notifications to their phone and
desktop- works great to pull people in as needed, without needing to distract
them with the noise of being in the room 24/7.

Our ops team is particularly in deep with chat. Instead of siloing everyone's
process off individually in SSH sessions, many operations happen in chat, so
everyone can learn and help out when diagnosing problems. They've built some
really fascinating tooling around the problems they face- if you're
interested, check out @jnewland's talk on ChatOps:
[http://www.youtube.com/watch?v=NST3u-GjjFw](http://www.youtube.com/watch?v=NST3u-GjjFw)

~~~
trumbitta2
I still don't get how do you manage to do security-sensitive things in
Campfire via Hubot :-/

~~~
mwarkentin
I've been working on adding configuration management via environment variables
at work. We allow setting configs via the command line, but it posts a
notification to our devops room in Hipchat showing who made the change, and
which config was changed, but not the new value.

This means that people can see _what_ is going on, without having full access
to the settings which may be sensitive.

I suppose you could also set sensitive credentials in a one on one chat with
Hubot.

~~~
ben_straub
I've seen this done by locking the room, too. That way, everyone in the room
when you make the change sees _everything_ (and can check your work), but it's
not recorded in the transcripts.

------
eliben
Nice presentation, thanks for sharing!

Favorite quote: "your product should be cutting edge, not your tech"

------
jcutrell
Love these talks from Zach Holman.

Brilliant work happening at GitHub, of course. Seeing the rise of the
"primarily responsible person" is huge for us in our small company - we've
found it's natural to work this way, without a label.

Keep on putting out these fantastic slides, Zach.

~~~
skrebbel
Admittedly I'm cynical, but how is a "primarily responsible person" different
from a lead developer? It feels a bit like you're just calling things
different to avoid manager-like names.

~~~
jbarnette
We use PRP in any situation where someone explicitly takes responsibility for
an outcome, no matter whether the outcome is some software or a clean office
or an accurate tax return.

~~~
dingaling
I'm intrigued as to what happens when no-one 'takes responsibility' for a non-
optional task. Like a compliance issue, or that tax return.

Who assigns the shit task to whom?

~~~
jbarnette
There is no such thing as a shit task.

Added: That's a little too glib, sorry. Let me try again: From my perspective,
if there's something that _needs_ to be done at GitHub there are a few
possibilities:

1\. It needs to be done and it's getting done,

2\. It needs to be done and it's _not_ getting done, or

3\. It's bullshit.

Cases of #3 become obvious pretty quickly. The best evidence: Searching for
ways to _make_ someone do it because nobody stepped up.

Cases of #2 can happen for a bunch of different reasons, but malice, apathy,
laziness, or incompetence are the least likely ones. The most likely: Not
enough hours in the day or not enough people with the knowledge necessary to
be worried. No matter the reason for #2, someone at GitHub who _is_ worried
will generally try to get others to share their priorities, by persuasion, by
hiring, or by prototyping.

Or occasionally by just jumping up and down and wailing.

------
wffurr
Anybody else made sad by

Open positions: Technical Account Manager

or just me?

Or it's like my company, and they don't bother to list positions for software
engineers because they're hiring those all the time.

------
hobonumber1
I'm a big fan of Zach Holman's talks and slides. He's a great speaker on these
topics. Looking forward to seeing the video up soon.

------
grandalf
I can't wait to try Github/REDACTED

~~~
Raphael
Which one? It's confusing to have 2 teams with the same name,
github:[redacted].

------
d0m
Funny, couple days ago I was wondering why zach stopped talking about the
github internal.. which I found illuminating and highly entertaining, and
then, a couple days later, boom an amazing presentation.

If I may add a question: How do you manage to keep a consistent design across
all teams considering that there's no manager and that most teams have
different designers?

~~~
holman
We keep a styleguide (which is actually public:
[https://github.com/styleguide](https://github.com/styleguide)) that we
loosely use across many products and apps. Beyond that, we have two libraries
(one CSS, one JavaScript) that we pull into projects that help us maintain a
general feeling of consistency.

Beyond that, designers get passed around the company a lot. Typically they can
implement the frontend quicker than the backend can be finished. This has an
interesting and unintended side effect of getting designers working all across
the company. I think this plays a large role in getting a lot of the
consistent feel across different projects: there's a lot of transfer of taste
across the company.

------
willejs
Dear github, please sort out your: \- Intermittent angry unicorns \- Slow data
transfer (150kb/sec?!) \- Poor support \- Expensive enterprise licence, why
don't you host a better service and i will pay for it?

~~~
WestCoastJustin
If you are so fed up with them, why not host an internal Git service with
gitolite for user access rules [1]. You could even add something like gitweb
as a web interface for git/gitolite [2]. Or, better yet, use something like
GitLab [3].

[1] [http://sysadmincasts.com/episodes/11-internal-git-server-
wit...](http://sysadmincasts.com/episodes/11-internal-git-server-with-
gitolite)

[2]
[https://git.wiki.kernel.org/index.php/Gitweb](https://git.wiki.kernel.org/index.php/Gitweb)

[3] [http://gitlab.org/](http://gitlab.org/)

~~~
RyJones
We used gerrit at amazon and my current employer and it seems to have plenty
of enterprise features out-of-the-box.

[https://code.google.com/p/gerrit/](https://code.google.com/p/gerrit/)

------
cenhyperion
Wow, I'm taking notes on the slide design. Looking forward to the video.

~~~
z999
He has an earlier blog post on how he designs his slides. Not extremely
informative but csn give you couple of tips on how to get that zach holman
look. Iirc it was in response to the how github uses github to build github
talk.

~~~
cenhyperion
Here's the blog post I think you're referring to.

[http://zachholman.com/posts/slide-design-for-
developers/](http://zachholman.com/posts/slide-design-for-developers/)

------
serf
[Video] coming soon.

So.. the topic is wrong?

------
lucasnemeth
That guy knows how to make good looking slides.

~~~
rurounijones
The one thing that would make it better would be to black-line border the text
I think so that it stands out on the black AND white parts of the slide.

------
fragmede
So... you guys are relying on a bot on a 3rd party chat system to do live-
deploys?

Does that scare the crap out of anyone else?

------
speg
How GitHub works is one of my favourite slides, looking forward see how things
have changed.

------
Maro
I was at the talk. It was great. Good job!

------
northisup
Not actually a video.

