
GitHub Down with a 503 - cheese1756
https://status.github.com/
======
dmoy
Random somewhat on-topic question. How does one measure ones technical
capability to keep up services that are more available and dependent than
third party solutions? I mean obviously starting out, almost everybody can
benefit from doing a lot of stuff not in-house, but at some point you'd be
able to do better yourselves. But then maybe for certain services from certain
providers, raw technical capability never catches up?

How do you know when that point is? Is there even data out there on how
frequently big providers go down? (Not just Github, but stuff like aws, etc)

~~~
bricestacey
By the time you're talking about 99.9% uptime there is only 9 hours downtime a
year so there isn't much wiggle room for failure. There is never a reason any
one person should assume they can do better than a third party because you're
likely sleeping 2920 hours a year. Basically, you're never good enough on your
own. If you care about uptime you're going to need to pay good money for it.

~~~
increment_i
But, hypothetically speaking as of now, could this be automated? And if so,
how?

~~~
bricestacey
In short, no, unless you think people replacing broken hardware or patching
bugs in your software can be automated.

Your question is too vague. Github is up enough that I don't care. However,
it's down enough I wouldn't want to not be able to deploy because it's down.
Therefore, I may mirror my repo somewhere else. That's easy because git is
decentralized. It's a lot cheaper than running some alternative that I
guarantee is always running. You can do this by simply pushing to a mirrored
remote branch.

If you're just interested in the subject, research high availability.

------
argonaut
Reminds me of the classic bumbling quote: "Why don't they just host GitHub
where they host the status page?"

~~~
cshimmin
"Why don't they make the whole damn plane out of the black box?"

~~~
altcognito
I'm always impressed that life support systems on star trek fail, but
artificial-gravity never ceases to work.

~~~
lloeki
Probably there are a handful of gravitons or something embedded right in the
floor, or there is massive inertia to the system. We tend to forget that not
all solutions require active power.

~~~
larrybolt
This does makes one wonder that if you need to make an app that has as little
downtime as possible perhaps the best solution is to port as much as possible
functionality client-side with client-side caching of data.

(theoretically, if I could cache every repo I visit using squid-proxy, even if
github has an hour downtime I can still access the repo's I visited in the
past, not perfect, but it's something!)

------
Aqueous
Alright...well, it looks like this whole programming thing was a bust. Was
good while it lasted, but yeah, uh, I guess I'm gonna go home, er, something -
so...see you guys later?

------
cclogg
Haha, I wonder how many of us came here because we couldn't push or pull
commits, so just taking a HN reading break... and boom the top post is about
Github!

~~~
mafro
When Github appears down, I come to HN to find out if it's affecting everyone
:p

~~~
YoukaiCountry
'Gee, github is down. Is it just me? Better check HN!' Exactly what I just did
too.

~~~
pwelch
This is what I do as we'll. :)

------
mrmondo
It's amazing how often github actually goes down - 90% of the time they don't
log it on their uptime / status page - it got so bad at one point I added it
to our Nagios monitoring to prove a point to our devs that they shouldn't rely
on external services for builds when our internal git(lab) server hasn't gone
down once this year.

~~~
copperx
How does one depend on Github for builds if git is, um, distributed?

~~~
lloeki
go get, npm install

BTW the official standing on dependencies in Go is to actually fork all deps
you need, possibly using submodules or even vendorizing by adding src/* to
your project VCS.

------
ereckers
I was waiting for this post :)

Let's see who the Saturday team is!

~~~
rsanheim
We have an amazing team of people who are available and on-call 24/7 for these
sorts of incidents.

~~~
ereckers
Oh I don't doubt that. I meant finding out who was lurking around HN on a late
Saturday early Sunday. The team at Github does great work!

------
viclou
[https://twitter.com/shit_hn_says/status/421116237792284672](https://twitter.com/shit_hn_says/status/421116237792284672)

------
martinvol
Exception percentage just got over 100%, new record! (?)
[http://i57.tinypic.com/4hphtd.png](http://i57.tinypic.com/4hphtd.png)

~~~
saurik
It is now up to 175% a couple minutes later (and 205% a couple minutes after
that). The issue seems to be that it is actually currently more like 10000%,
and is being averaged down throughout the month to a mere 100-200%. What is
this even measuring?!

~~~
helper
If I were implementing something tracking the rate of exceptions a reasonable
thing to measure it against is the total number of requests. In such a system,
if each request generated multiple exceptions you would have >100% exception
rate.

------
aroman
I was wondering why my `bower install` was failing... :/

~~~
qntmfred
Doesn't bower install from a local cache if network request fails? npm on the
other hand...

per docs at [https://github.com/bower/bower](https://github.com/bower/bower)
oh....

~~~
aroman
Yeah it does pull from cache, but I didn't have the packaged cached yet.

As for the docs, it is somewhat crazy to think how much documentation is
hosted by GitHub these days...

------
wildmXranat
Yep. All metrics down and exception rate is up gentlemen:
[http://imgur.com/Ncm5Rrx](http://imgur.com/Ncm5Rrx) . I just noticed when
SQLMap install was hung and every tarball/zip download on their site also
leads to Github , oy vey!

~~~
AYBABTME
At least webhooks are becoming faster..!

------
robbles
Could Github store mirrors of all repositories on some third-party service,
like S3?

Even if they were only updated every 24 hours and had a limited history for
each repo, it still seems like it would be a really useful fallback that they
could put up when things like this happen.

~~~
eli
It's actually not _that_ hard to set up your git to push to both Github and
another repository at the same time.

~~~
robbles
Sure, it's easy to do with your own code. But what about apps that don't
vendor their dependencies? If you need to pull in third-party dependencies
when you deploy or test, it's still possible to maintain your own mirror, but
it would be nice if you could depend on the code being "up".

------
ChiChou
Somehow I opened the GitHub Status Page subconsciously early this morning,
then everything suddenly became 503 in less than 10 minutes!

I still have the tab opened now, saying "All systems operational"...

~~~
ChiChou
Seems like it's back now. (3:13 UTC)

------
mattkrea
Why have they gone down so often in the past few months? Is this another DDOS?

I've personally wondered about their stack but I imagine that has little to do
with that--mostly Ruby though right?

~~~
sinaa
It seems to be MySQL related this time:

"github.com is currently unreachable. We are investigating mysql cluster
issues."

------
4dl0v3-p34c3
I hope it was just an error or something. Nothhing political, like this:

[https://gist.github.com/kyledrake/e6046644115f185f7af0](https://gist.github.com/kyledrake/e6046644115f185f7af0)

More info:

[http://www.theverge.com/policy/2014/5/9/5699510/web-
hosting-...](http://www.theverge.com/policy/2014/5/9/5699510/web-hosting-
company-puts-fcc-in-slow-lane)

~~~
hayksaakian
the gist link is also a 503

------
papercruncher
Tempted to deploy via rsync on my laptop

------
co2
I'm wondering if this will give bitbucket a boost in unique visits.

------
joshdotsmith
Right in the middle of a major push to production. Awesome.

~~~
akerl_
It's too bad git requires a central server.

~~~
AYBABTME
How is this a useful comment? Everybody knows that Git is distributed.
Everytime Github's down, somebody feels the urge to remind everybody else
about that.

Github's not only Git. Lots of services integrate with Github. Stuff like
continuous integration, continuous deployment. If your system is built on
those things, Github being down will prevent you from deploying.

What's the alternative, replicate everything in-house/self hosted? Should
startup stop using third party service providers?

~~~
akerl_
If your workflow involves services that integrate with GitHub, you should have
a plan for when GitHub is not available. You've got your source code
elsewhere, you should have docs and procedures for running the tests in your
own dev environments and for utilizing whatever staging/deploy process you
prefer without GitHub as an intermediary.

I continue to post comments like that because while I love GitHub, watching
the community put all their eggs in GitHub's basket, especially when GitHub
lives on top of a tool designed to avoid SPoFs, is concerning.

~~~
hamburglar
100%. GitHub is amazing. GitHub is a wonderful tool. GitHub is also just a
service cooked up by a handful of jokers who got enough funding to make it
happen, and they do a reasonable job of keeping it running most of the time. I
wish them all the best, and yet making my own company's infrastructure reliant
on it is not something I'm comfortable with. Sometimes they're going to blow
it, and that's ok. If them blowing it means my business is affected, _I 'm_
the dummy.

~~~
skrebbel
> _jokers who got enough funding to make it happen_

While I agree with the general gist of your comment, GitHub was entirely self-
funded up until about a year ago or so. they didn't "get" funding, they made
real money and reinvested it and grew.

~~~
hamburglar
Fine, bootstrapping jokers are my favorite kind of jokers. No offense
intended. My point is it's just a small company with no SLA, express or
implied. It could have been anybody with the same great idea, and just because
they had a great idea doesn't make them automatically reliable enough to pin
your infrastructure on.

------
danielhellier
Curious, which chart lib is GitHub using for those charts?

~~~
cheese1756
They make calls to d3 in the chart snippet, so I assume it's that. Here's the
relevant code from application.js:
[http://pastebin.com/ApL9SxXs](http://pastebin.com/ApL9SxXs)

------
mhahn
all the overachieving engineers working on a saturday...

------
StepR
Oh damn. The Github team is in for a rude awakening...

------
nodesocket
> We are investigating mysql cluster issues.

------
simplon
Yeah seems so.. [https://status.github.com/](https://status.github.com/)

~~~
StepR
Thats what the link above goes to

------
OutThisLife
It's all over now.

------
thamaraimedia
GitHub now available.

------
iancarroll
and right when I wanted to clone something. damn.

------
zcs
It's back!

------
nxbtch
you are really the early bird.

~~~
cheese1756
I was setting up a new repo, and then everything stopped working. Lucky
timing, I guess.

