

Bye Bye Github - razerbeans
http://blog.eizesus.com/2010/2/bye-bye-github-7-2-2010/

======
keyist
"a real serious business would have created an infrastructure that will
provide paying customers a better redundancy for their accounts"

He should apply the same argument he makes against Github to his own business.
Putting together a Gitosis or similar setup mirrored to Github would have
given him and his paying customers the necessary redundancy to deploy when one
of the hosts is down.

EDIT: Figure while I'm here and it's not OT I should give a well-deserved
shout-out to Gitosis which has made my life a lot easier the past two years or
so:

Gitosis setup is a one-off, very low maintenance install (you add repositories
and users by committing config changes and public keys to a git repository --
hardly anything out of your regular workflow). Once set up, mirroring to
github or another service is a one-liner in your post-receive hook.

~~~
mrshoe
To borrow Jeff Bezos' analogy from Startup School, should all small businesses
who pay a utility company for electricity also have enough generators to run
their business should the utility company fail temporarily?

The raison d'etre for companies like Github is that small software companies
can't afford to waste a lot of time building infrastructure. If they want to
survive, they need to focus on their core competencies, their customers, and
their products. If outsourcing infrastructure functions to Github doesn't
allow you to do that, then there's not much reason to pay Github.

~~~
vulf
The electric company analogy is a good one, but I don't think you're looking
at it correctly.

No service, not even electricity, has 100% uptime. If thinks are _that_
mission critical, you need a backup plan. Should you disconnect from the grid
and generate your own electricity? Probably not. But should you expect that it
might go out at any time? Yes. If you are not running a UPS than no one is to
blame except yourself. If things are so critical that a UPS won't last long
enough, then you should have a generator backup.

In contrary, what would the poster have done if his power or internet had gone
down when he needed to deploy? Would he be cancelling those services? For some
reason I don't believe he would.

The real key to all of this is not a question of avoiding downtime, it's
learning from it and avoiding a repeat of the same incident. I think GitHub
has done a great job of this, and they're very clear about what happened and
what they've done to avoid it in the future. The poster, however, fails at
this point. Not only does he admit he doesn't have a backup plan for deploying
when GitHub is down... he flat out states that he knows what he could have
done, and REFUSES to do it. Instead he's just going to cut and run, thinking
he'll find another service that somehow is somehow immune to unexpected
downtime. I wish I could live in that world.

~~~
mechanical_fish
Moreover: Nothing comes without cost. If you want a service like GitHub but
with higher reliability, you are going to pay. Possibly in money, possibly in
ease of use, possibly in the time it takes to _identify_ such a service (it's
really hard to gather reliable uptime data, especially in a field that is
constantly evolving).

And, all things being equal, I'm not anxious to pay more money for a more
reliable Github. _Because it's git, people._ Why pay for five nines of uptime
when you can just mirror your repo? Every five minutes, if you like? With a
one-line cron task? _One of git's most basic functions is to make an efficient
mirror of itself._

~~~
vulf
Indeed, GitHub's terms even state "GitHub does not warrant that ... (ii) the
service will be uninterrupted..."

If uptime was a serious factor for you, you'd probably pay for an SLA to
ensure uptime and compensation for downtime... and a SLA wouldn't be cheap.

~~~
vidarh
One of our customers wanted an SLA with more teeth. They themselves with a
straight face proposed an SLA that would give them ca 8 GBP in credit against
hosting fees per _day_ of downtime. This is a multi-million pound business. We
said yes, of course.

We'd happily negotiate something with _real_ teeth, as I'm sure most service
providers would, and I'm sure Github would too if a big customer pushed for
it.

But you're 100% right - if we did it'd be expensive, because we'd have to turn
right around and insure ourselves against the risks incurred if the amounts
were remotely serious, and we'd pass that insurance cost straight on.

------
patio11
Ignoring the fulminating a bit and concentrating on the substantive issue,
there is a real leap in expected level of service once you start taking money
from people, and again when you start hosting Mission Critical apps. (What an
overused buzzword -- but a _true_ overused buzzword!)

Saying "He shouldn't have had anything Mission Critical on Github, he should
have designed his infrastructure competently" is besides the point. Properly
designing a fault-tolerant workflow involving git isn't something which is in
the easy reach of everyone, and some people would prefer to pay money than to
deal with the hassle of doing it themselves. (And trust me, it is a hassle. I
have my own gitosis server because I had a business necessity to host my OSS
on my own domain. Holy bleeping heck I lost a day of my life to getting that
working. That's close to $1,000 of engineer time replicating something very
close to what every git hosting company offers for $9 a month.)

People who don't have the skills and don't have the desire to set up a fault-
tolerant git infrastructure pay other people to do it for them. Other people
being... well... Github. That means that Github has acutely more sensitive
uptime demands than somebody's blog on OSS topics or even paid services in
less critical contexts. One reason my business is in a less critical context
was because I knew that the constraints I was working under made it
unreasonable for me to offer customers the assurance that they could build
their businesses on me.

~~~
dmoney
Does Github offer the assurance that customers can build their businesses on
it? If you want an SLA you will probably pay more than $9 per month (beyond
the implicit SLA of "I'll take my business elsewhere if somebody else has
better uptime for the same price").

~~~
patio11
SLAs are largely meaningless pieces of paper invented by folks who either
think that downtime can be wished away by writing the number 9 enough times,
or largely meaningless pieces of paper meant to quiet the discontent of non-
technical stakeholders by pointing them to a suitably impressive numbers of
9s.

~~~
mahmud
I have just cut and pasted that into a presentation I am making tomorrow :-)
Page 2.

~~~
patio11
I'd love seeing a copy of that if you don't mind. (Incidentally, should any of
you ever want to do this to a quote on HN from me, feel free. I trust your
judgement regarding whether it is substantative enough to merit attribution.)

------
risotto
Git is decentralized. There's _no way_ Github can affect your ability to
deploy. Code can only be pushed there meaning it was written and exists
somewhere else.

There's also no way to run your own private SSH or git server without paying a
minimal monthly fee. Any such server or Internet connection will go down
occasionally.

Nobody is going to stop you from finding an alternative that suits you better.
Good luck.

~~~
n8agrin
Thank you, you are the only poster yet that seems to get it. I agree, the
point of git is that you already have your entire repo. Who cares if your
remote host is down? Deploy using your branch and when github (or whatever)
comes back up you push your changes to github. Done. I don't think of Github
as a centralized server, but as a place to search and share code.

~~~
chriseppstein
This is true, but it's a hassle. The one time I needed to deploy and github
was down, I had to read a link on their website and set things up. It was a
real drag and delayed my launch by at least 15 min!!!

------
eli
_shrug_

Well, I kinda agree, but if github is really a mission critical piece of your
company, then perhaps you should be hosting your own repository.

Or at least mirroring it somewhere? Isn't that half the point of using git
over svn?

~~~
timdorr
With Git, every copy _is_ a mirror.

------
bretpiatt
This is really harsh and it sounds like he's looking for a scapegoat on why he
is behind on client projects, "i delayed 3 client deploys in the past week due
to Github’s downtime".

GitHub is very transparent about their issues and from looking at their blog (
<http://github.com/blog/597-a-note-on-the-recent-outages> ), "Following three
months of near 100% uptime, we’ve just been through three major outages in as
many days."

The outages weren't "days long" so those delays in deploying a client's
project shouldn't have been significant. If the client expected a deployment
during a short window it is irresponsible for the development company to keep
the code only in a single location. I cover this in a post about availability
on my blog ( [http://www.bretpiatt.com/blog/2009/10/03/availability-is-
a-f...](http://www.bretpiatt.com/blog/2009/10/03/availability-is-a-
fundamental-design-concept/) ).

------
forsaken
We just had a discussion about this a couple of hours ago. If your deployment
depends on an external service, your deployment is broken.

You're using git. Have a fabric script that pulls down the remote repositories
that you need, and rsync's them to your servers. This works locally or on a
remote box. If github is down, pass in local directory to sync instead. This
way you can always push.

He built his infrastructure wrong, and is trying to blame someone else. Sorry,
but you are just as much at fault, if not more, than github.

~~~
zacharypinter
I don't think that's an entirely fair argument to make.

For example, I don't think it's unreasonable to write a unit test that expects
an SMTP server to be available. Maybe doing so would break the deploy if the
SMTP server ever went down, but at that point you have two options:
rearchitect your deploy, or find a more reliable SMTP server to test against.

If the author would rather spend his time and money looking for a more
reliable git host than rearchitecting his deploy to account for service
failures, why is that so bad?

~~~
vidarh
It is so bad because it fails to take into account that there are multiple
failure points involved (his git host, his internet connectivity, multiple
transit providers and routers beyond either his or his hosts control,
electricity supply etc.) and that getting a more reliable git host won't even
address one of them - no hosting will ever achieve 100% uptime, if only
because of eventual human error.

He'd be better served first addressing the significant flaw in his overall
process of not having a fallback that bypasses as many failure points as
possible.

I.e. why does he not have a constantly updated clone of the repositories
_locally_ at the provider hosting his actual sites, so that he can deploy as
long as he can get hold of an internet connection somehow and is able to
connect to his hosting environment?

This bypasses the local electricity supply, his office internet connection,
the transit providers between him and Github, etc., because there are many
easily accessible ways of getting an ssh connectio in a pinch, and the
remaining dependency is his production hosting, without which deployment
wouldn't work anyway.

It would be a simple, cheap and prudent safety mechanism if being able to
deploy immediately is business critical for him. It's a matter of a single
command in a crontab on one or more of his production servers.

------
eladmeidar
I think i need to clear it up a little bit. (Yeah, it's my post).

I agree that the post came off a bit too harsh maybe, but as far as
reliability for a payed service goes, github was yet to provide a decent
level. Ever since Github outages started (a while ago, even on EY's age) i
decided to push into 2 separate locations and therefore giving Github a
further chance (2nd location is another paid account on Unfuddle.com) but the
delay on deployments was due to simple need on our side that resulted in the
need to push another change to github (updating a vendored private gem) while
this process may probably not be perfect it does not remove any responsibility
from github as a payed service to provide a better way of supporting payed
accounts.

i will review my internal processes further more to see if there's a better
way of doing them, but as far as continuing a payment to an unreliable
service.. that part is over.

~~~
peteforde
I think when companies like 37signals talk about firing clients, you might be
the guy they had in mind.

Just saying... I suspect that they can do fine and provide "four nines"
without your $7/month contribution.

~~~
patio11
"Four nines" is one of the reasons SLAs are pretty meaningless. That's an hour
of downtime every year. One unplanned outage blows that.

Google's flagship product (search) probably breached four nines last year for
some users (they had 20~40 minutes of effective downtime with that one "All
sites have malware" incident) and they're much, much better than you, I, or
Github will ever be.

------
daeken
He makes some decent points about Github's response to their reliability
issues, but that's about where it ends. This rant can be summed up as: Github
is unreliable and they should work on this, pushing free users to the back
burner in favor of their paying clients. Whether or not you agree with that,
the rant was a bit unnecessary.

~~~
vulf
How would they do that though? If you look at the details of what cause the
outage, it was not a matter of free vs. paid users. Even if they added extra
redundancy for paid users that they didn't give free users, that wouldn't do
any good in the case of a site-wide failure, which is what they had.

For that matter, I don't think they've ever had an issue that could be
isolated to affect only free users, have they?

~~~
zacharypinter
For Outage #3 they said "After inspecting the HTTP logs, we identified a
Yahoo! spider that was making thousands of requests but never waiting for
responses." Spidering isn't an issue for the paid private repos since they
have nothing public to index. However, the web access to private repos went
down due to the spidering of the free user pages.

------
zacharypinter
I think this article is being a bit harsh and overly dramatic. However, he
does have a point in that the github folks should probably try to architect
things such that their paid service is isolated from any mishaps that come
from the free service.

------
amock
I don't currently have anything hosted at Github, but I use them regularly to
check out other people's code and their uptime history is disappointing. I
know that they always explain what caused the downtime and I'm glad that they
do. Explain what's wrong isn't enough, though. It seems like they have a
poorly designed system that can be brought down by almost any piece failing.
Their latest outage makes it sound like they don't know how to build a fault
tolerant system at all. They have the parts there, like DRBD, but when things
go wrong everything breaks anyway. It's like making backups but never testing
them so that when you need them it turns out that they can't actually be used
to restore anything.

It seems that they have the part that most companies miss, the communication,
but that doesn't make up for the lack of reliability.

------
dlsspy
This kind of feels like another "don't know how to use my tools" kind of post.

I was doing some critical work during the last time. I had to collaborate with
another developer and he was unable to push through github. So we exchanged
code a different way.

I was also doing EC2 tests of the code we were working on during this window.
I didn't have trouble getting that deployed (via git, no less) during this
time.

It's easy to do the lazy thing and rely on centralized systems when it's
_completely_ unnecessary just to avoid a small amount of work to _greatly_
increase the reliability of your platform.

------
tdmackey
Yes, it is pretty bad for github to have so many problems in so few days and
if he isn't happy with the service he should leave.

However, seems like he doesn't really understand how to use Git if github
being down is barring him from doing his job or deploying his work. Having a
single point of failure while using a distributed version control system is
incompetence on the developer more so thann it is a problem github being down.
Github provides you a lot of nice features and fancy web based interfaces and
some social aspect to coding, but to rely solely on it shows a lack of
understanding and removes many of the benefits git provides a developer.

------
pmarsh
Not sure what the big deal is.

Customer doesn't feel like the price they are paying is worth the value and
leaves.

Only difference is that there are alternatives to this "critical service."

Had this been a rant about the electric company going down in the snowstorm no
one would have cared because it was understood these things happen.

Have an absolute need for power no matter what? Better get yourself a backup
generator. Just because you pay for a service doesn't mean it won't go down.
Just means you should expect them to fix it ASAP.

------
slig
I've been very happy with repositionhosting.com uptime and support.

------
jpcx01
I pay github for a private repo I rarely use, however I really depend on them
for my development as all the open source libraries and documentation I use
are stationed there. I don't see switching to something else as an
alternative.

So what can we do to help them keep their servers up for the community's
benefit?

------
thomasfl
Just use gitorious <http://gitorious.com>. If you are not satisfied with the
hosting, install locally <http://blog.alexgirard.com/git-gitorious-on-ubuntu>

------
mscarborough
Ouch, that font is hard on the eyes. I have minimum font sizes for
readability, but even turning it down, jeez.

Thanks again to the Readability marklet!

------
jrockway
It's the Internet. Everyone has downtime, even Amazon.

My only complaint about Github is how my profile is turned into a giant ad for
github when a non-logged-in user visits it. I think it's really underhanded to
use a paying customer's profile page to convince random people to sign up for
github. If they want to, they will figure it out, just like the other million
users.

~~~
kneath
Would you mind clarifying that for me? We definitely do not intend to do that
in any way possible. For example, here is my profile page, logged out:
[http://share.kyleneath.com/captures/kneath_s_Profile_-
_GitHu...](http://share.kyleneath.com/captures/kneath_s_Profile_-
_GitHub-20100207-181426.jpg)

There shouldn't be any excess GitHub branding apart from the obvious bits
(header, footer, terminology).

~~~
freetard
On your screenshot I see two links for login and pricing on the header and
right down to it.

~~~
aaronbrethorst
Your username is rather priceless in the current context, btw :)

Rackspace costs money. Servers cost money. Bandwidth costs money. Salaries
cost money.

There's nothing shameful about asking users for money. Putting a 'Sign Up'
button or two on the page is just good design.

Would you prefer they buried a PayPal donation button on their About page?

I'm also curious about why jrockway thinks a Sign up button is underhanded.
Maybe if there was some sort of bait and switch involved, I might see their
point. But, the 112x23px button pretty clearly says "Pricing and Signup". Of
the 920px x 3783px of content on kneath's profile page, that 112x23px
represents 0.07402% of the page.

Plus, it may be _your_ profile page. But it's your profile page on Github's
server. If you don't like it, why not build a custom HTTP front end to your
privately maintained git repos?

~~~
vulf
Users can add a pledgie donate button to their repos:
<http://github.com/devinross/tapkulibrary> Looks like it's bigger than that
signup button too...

------
Groxx
The large amount of grammar and spelling errors make me question if the author
is at all interested in a "real serious business". And WTF is up with the
horrendous comma usage? It's similar on his Nautilus6 and LinkedIn websites,
too.

Also, how precisely did temporary downtimes prevent deploys? At best, it
appears they would've been delayed by a couple hours, and the great part about
Git is that you can work around something like that pretty easily. Granted,
downtime is annoying, but his extended difficulties seem to largely stem from
his own mistakes, and for $7/month he's not exactly paying top dollar to
guarantee uptime.

Learn too spell: <http://iampaddy.com/spell/>

~~~
eladmeidar
Like i noted on one of the comments on the post, the need to wait a few hours
was due to a last-minute change we had to push.

yes, i do need a better process (even if this case WAS a very specific case
that is not likely to happen again) but still, if i pay i want to get the
service i pay for.

Github had outages since ever, and although they do seem to do the right thing
and make it better, it seems that they are aiming at the wrong direction.

And about the spelling/grammar/whatever errors, i'm just not a native english
speaker. Don't think that was the real issue in the post but whatever, one day
i'll learn to spell.

~~~
vulf
The spelling/grammer issues just add to the feel that your post was a pissed
off rant. My own typing goes to crap if I'm in a rage. The key is to wait a
period of time to cool off and go back over the post and fix the errors before
you send it, that's all.

~~~
eladmeidar
i'll keep that in mind. (hopefully no "for next time" is required)

